Storage medium for storing a signal having successive nonuniform images for subsequent playback and a method for forming such a signal for storage on such

ABSTRACT

A storage medium has successive nonuniform images stored thereon for retrieval and presentation by a playback device to an eye of a passive viewer, each successive nonuniform image having a highly detailed component and a lesser detailed component, the successive nonuniform images for simulating percepts of an observer of an object space wherein a position of the highly detailed component is changed in selected successive nonuniform images such that a visual axis of the eye of the viewer with respect to a field of view of the selected successive nonuniform images is drawn to passively follow the position of the highly detailed component so that the viewer experiences percepts as if looking through the eyes of the observer of the object space. A method for producing successive nonuniform images for storage on a storage medium is also shown.

RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.10/948,809 filed Sep. 22, 2004, which is a continuation of U.S. patentapplication Ser. No. 09/772,016 filed Jan. 29, 2001 as acontinuation-in-part of U.S. patent application Ser. No. 09/524,491filed Mar. 13, 2000 (now U.S. Pat. No. 7,453,451 issued Nov. 18, 2008)and claiming priority from U.S. provisional application 60/124,642 filedMar. 16, 1999 (now U.S. Pat. No. 6,798,443 issued Sep. 28, 2004), whichapplication Ser. No. 09/772,016 is also a continuation-in-part of U.S.patent application Ser. No. 08/794,122 filed Feb. 3, 1997 (now U.S. Pat.No. 6,181,371 issued Jan. 30, 2001), which is itself acontinuation-in-part of U.S. patent application Ser. No. 08/452,510,filed May 30, 1995 (now U.S. Pat. No. 5,734,421 issued Mar. 31, 1998)

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates to providing light for forming images and, moreparticularly, to providing light for forming images for a passiveviewer.

2. Description of Related Art

Still photography, motion pictures and television were influenced by theway artists represented physical reality in paintings, as if through awindow. A highly detailed perspective image is provided, typicallywithin a rectangular frame. All provide highly detailed images whichinduce the viewer to cooperate with the cameraman's “vision” by assumingthe artificial perspective of the representation. The viewer is enabledto deliberately suspend disbelief that the images themselves are not areal object space. The degree to which the viewer is thus enabled isinfluenced not only by the image resolution but by the field of view. Itis usually thought desirable to increase both. For example, very highresolution commercial television standards have been formulated forincreasing image quality. Such approaches typically increase the numberof horizontal lines scanned to a number significantly greater thanpresent standards. Larger format movie film such as 70 mm has been usedto increase detail. Also, panoramic movies, e.g., “Cinerama” increasedthe field of view to increase realism. Various stereoscopic televisionapproaches have also been conceived or developed to increase realism.

All of these traditional media take a rather objective view of thephysical world. The image is framed by a window through which the viewercan gaze in any direction “into” a representation of an object space.Events are presented in both movies and television in a series ofdifferent action scenes in a story line which the viewer can observefrom a stable and seemingly quasi-omniscient point of view. The vieweris led to take what appears to be a view of the world as it really is.Yet the choice of image and its perspective is picked by the creator ofthe image and the viewer actually assumes a passive role.

A sensorama simulator was disclosed by Heilig in U.S. Pat. No.3,050,870. The senses of an individual were stimulated to simulate anactual experience realistically with images, a breeze, odors, binauralsound and even motion. Heilig also disclosed a stereoscopic televisionin U.S. Pat. No. 2,955,156. This also was passive.

“Virtual reality,” in an electronic image context, goes even further inthe direction of increased realism but enables the viewer to take a moreactive role in selecting the image and the perspective. It meansallowing a viewer's natural gestures, i.e., head and body movements, bymeans of a computer, to control the images surroundings, as if theviewer were seeing and moving about in a real environment of seeing,hearing and touching. Due to the myriad of possible actions of theviewer, a corresponding multiplicity of virtual activities needs to beavailable for viewer choice. This would represent the ultimate inartificial experience.

A user of a “virtual reality” device will typically don a head-mounteddisplay which provides images of a virtual space that are matched to thesensed position and orientation of the head of the user as the usermoves his head in space and time (e.g., the x, y, z position of the headand/or the roll, pitch, yaw attitude of the head). For example, aFakespace BOOM3C is a Binocular Omni-Orientation Monitor that providesvisual displays and tracking integrated with a counterbalancedarticulated arm for full six-degree of freedom motion (x, y, z, roll,pitch, yaw) and provided by Fakespace, Inc., 241 Polaris Ave., MountainView Calif. 94043. Another example would be a wireless magnetic motioncapture system such as the STAR*TRAK of Polhemus Incorporated of 1Hercules Drive PO Box 560 Colchester Vt. 05446. It providessix-degree-of-freedom (position and orientation) data from up to 32sensors capturing data at up to 120 Hz.

The images for such devices are created by a computer program with theassistance of pre-stored image information that is retrieved accordingto the user's head movements and presented to the user's eyes. Theuser's head may be coupled to the display. The aim is to presentpanoramic images covering a wide field of view in order to immerse theuser in an artificial reality with which he can interact, as if real.The degree of artificiality need not be total and can instead constitutean “augmented reality” with some artificial objects or symbolssuperimposed or interposed within the real world as viewed with asee-through, head-mounted or head-coupled display.

These advances take advantage of converging technological developmentsin telecommunications including broadband services, projection opticsfor head mounted and head-coupled displays (including virtual retinaldisplays), the ever-increasing computational power of image processingcomputers, specialized sensors such as gloves designed to sense hand andfinger movements, exoskeletons, and the like. They can be expected tolead to exciting interactive games and other new forms of interactiveexperiences within virtual worlds.

This new paradigm represents a very great improvement over the presentimaging technology. It joins immersion to interactivity to increase thelevel of experience. It is now being applied to gaming applications andothers such as virtual museums, architectural and interior designmockups, facility tours, “aircraft” rides and the like.

The new paradigm would likewise seem to hold the potential for animprovement over the old ways of traditional entertainment such asdrama, comedy, documentaries, and the like. By joining immersion andinteractivity, the user would be enabled to enter a completely new realmof artificial experience. The user would be given a very high degree offreedom, under his own volition, to navigate in the virtual world and toparticipate in completely new forms of such entertainment, where theuser's own actions influence the sequence of images and audio provided.

Traditional entertainment applications, on the other hand, such asdrama, comedy, documentaries, and the like, have not yet been exploredby these new technologies. This could be because the traditionalapplications have usually been presented for passive enjoyment by theviewer. Even though immersion would provide a better experience,interactivity would be contrary to these known traditional entertainmentapplications, such as storytelling, where people like to relax and bepassively led through stories and participate vicariously. Anotherobstacle would seem to be that the level of complexity of the possiblealternative scenarios, depending on the user's actions, would need to behigher in the traditional arts than for the more predictable andmechanistic art of gaming.

For all these various kinds of virtual reality applications, thecreation of many possible scenarios for viewer selection creates amassive demand for electronic image storage space and there is also theproblem of a disconcerting time lag between the viewer's action and theresponse of the imaging system. These problems make this emergingtechnology hard to achieve using presently available hardware. Thesoftware task is equally daunting.

SUMMARY OF THE INVENTION

An object of the present invention is to provide a new method and meansof providing light for forming images for a viewer.

According to the present invention, a method of providing light from alight source at an orientation of the light source to an eye in a headof a viewer for formation of images in the eye, comprises the steps of:

providing the light from the light source for the formation of imageswith a changing point of view; and

changing the orientation of the light source in correspondence with thechanging point of view for guiding the head of the viewer in acorrespondingly changing orientation for viewing the images with the eyein the head of the viewer at the changing orientation of the lightsource and from the changing point of view.

The present invention may be carried out by apparatus, comprising:

a light source, responsive to a light control signal, for providinglight for viewing images by an eye in a head of a passive viewer; and

a light source actuator, responsive to a head guide control signal, forcausing the light source to execute attitudinal movements for emulationby the head of the passive viewer.

The actuator may be a robot configuration selected from the groupconsisting of Cartesian, cylindrical, spherical, and articulated robotconfigurations.

The images may but need not be created by means of one or more camerasassociated with a cameraman, for example, on the head of a cameraman.These are provided, according to the invention, for passive perceptionby a viewer whose head movements are guided by a motion-controlled headguide that is actuated in such a way as to emulate head movements of thecameraman in synchronism with the images actively sensed by thecameraman. The “cameraman,” if there is such, can but need not have oneor more cameras mounted on his head and the direction of his head withrespect to a selected reference frame is monitored; head monitoringsignals are stored in association with individual images picked up bythe head-mounted camera or cameras. Such images are provided “live” orare played back to the passive viewer by way of a display fixed on or inthe head guide, e.g., by way of a headup display fixed to the headguide. The motion of the head guide is controlled with respect to theindividual images by retrieving the previously stored head monitoringsignals in synchronization therewith. The head of the passive viewer isurged by the controlled movements of the head guide to execute headmovements emulative of the monitored motions of the cameraman at thetime of image acquisition.

Simulated active percepts, according to the present invention, permit aviewer to experience percepts passively, as if inside the head ofanother person. This “other person” is the “one” controlling theacquisition of the percepts experienced by the passive viewer. Eventhough the images presented to the passive viewer may be panning aboutand changing perspective at the whim of the “other person,” e.g., thecameraman, the passive viewer has those images presented to his eyeswhile his head is also urged to move in the same direction as that ofthe cameraman's head so that it is directionally coordinated with theimages viewed by the cameraman, as if he were viewing them himself,through his own eyes.

It should be realized that cameras are not needed and the images can becreated by means of a computer workstation or even by known animationtechniques coupled with computers and/or cinematography. In that case,the head movements can be preplanned rather than sensed.

There can be a large number of passive viewers with their ownmotion-controlled head guides. These can be embodied in second-hand(passive) experience simulators, e.g., in the form of self-containedbooths each with a multi-degree of freedom head guide for connectionwithin. The viewer's head guide may be actuated in any number of degreesof freedom, as a matter of design choice, to exert some minimum degreeof mechanical head guidance control with just a few actuators or canprovide a full complement of actuators, e.g., providing control in sixor even more axes. A booth can be for home or arcade use, for example.Such a viewer enters the booth, sits down and mechanically couples hishead to the head guide. E.g., the display may be a panoramic displayfixed in the wall of the booth or may be a helmet mounted display, asknown in the art. The invention need not be embodied in a booth. It canbe desk mounted or mounted in any convenient way.

The images provided to the passive viewer's eyes can be varied in theirapparent distances, e.g., by changing the focus of the optics in aneyepiece of the light source. In this way, the accommodation of the eyesof the viewer can be urged to follow the changes in focus at differingdepths within the image space.

The invention may be made even more like a re-experience of experiencesof another, according to another aspect of the present invention, byeffectively controlling eye movements of the passive viewer in such away as to emulative of eye movements of the other, e.g., the cameraman.This can be done in a nonintrusive way by presenting nonuniform imagesemulative of the human fovea, e.g., with nonuniform resolution,nonuniform dynamic range, a small colored area in an other-wisewide-field black and white image, nonuniform image informationalcontent, nonuniform image concentration, nonuniform brightness, or someother equivalent nonuniform images to the passive viewer, made so as todraw the viewer's attention to an accentuated area, wherein such areamoves about between successive images presented within the field of viewof the viewer. In this way, not only the head of the passive viewer hasits motions guided but the eye movements are guided as well. So thepassive viewer can have his head guided to be directed in one directionwhile the attention of his eyes is drawn or guided in another direction.In this way, the passive viewer feels even more like he is undergoingexperiences of another, e.g., the cameraman. Such images can be createdby monitoring one or both eyes of the cameraman and causing the imageinformation gathered by the cameras to be encoded in a nonuniform waysuch as by having finer scanning in a small area dictated by where thecameraman happens to be looking at a given moment with the rest of thefield scanned coarsely.

Furthermore, when coupled with the previously described changing focusfor changing the apparent distances of the images, the foveal viewingaspect of the invention can be used to “control” the exact point offixation to which the passive viewer's visual attention is directed,thereby establishing a sequence of fixations at various points atvarious depths in the image space with correspondingly differingaccommodation and convergence of the viewer's eyes.

Such simulated active percepts may be presented “live” or may be storedand retrieved from storage and later presented for passive perception. Abooth can, for example, be provided with a video cassette recorder toplayback the image and head guide control information. The storedimagery could even be downloaded or provided “live” from the Internet.

In the case of stored simulated active percepts, according to theteachings hereof, since there is only one set of images to store, themassive memory demand problem of the prior art of “virtual reality” issolved. Similarly, for the “live” case, since the simulated activepercept is provided at the same time as it is created, there is nostorage requirement at all, i.e., beyond temporary, “on-the-fly” storageneeds.

Moreover, by providing simulated active percepts for passive perception,there is no longer any time lag or latency problem as is presently thecase for known virtual reality applications. Since the simulated activepercepts induce the passive viewer to emulate those physical actionswhich caused or would have caused the simulated active percepts, thehardware need not be faster or as fast as the viewer. In fact, it may bemuch slower. Although the viewer is relegated to a passive role, thenovelty and richness of the “virtual reality,” immersive experience morethan compensates in opening a whole new world of opportunity forrepresenting reality.

These and other objects, features and advantages of the presentinvention will become more apparent in light of a detailed descriptionof a best mode embodiment thereof which follows, as illustrated in theaccompanying drawing.

BRIEF DESCRIPTION OF THE DRAWING

FIG. 1 shows apparatus for providing light to an eye in a head of aviewer for formation of images in the eye, according to a first aspectof the present invention.

FIG. 2A shows a suspended embodiment of the apparatus of FIG. 1.

FIG. 2B shows a desktop supported embodiment of the apparatus of FIG. 1.

FIG. 3 shows various robot embodiments of the apparatus of FIG. 1.

FIG. 4 shows one of the robot configurations of FIG. 3 in more detail.

FIG. 5, according to a second aspect of the invention, shows a cameramanin an image space for gathering light in cameras for transformation intothe light control signal of FIG. 1.

FIG. 6 illustrates various ways, according to a second aspect of thepresent invention, to create the light control signal and the head guidecontrol signal of FIG. 1.

FIG. 7, according to an embodiment of the second aspect of the presentinvention, shows a helmet for a cameraman in an object space having atleast one camera and various sensors for at least monitoring headattitude and a signal processor or encoder for providing an encodedsignal to a decoder, according to the first aspect of the invention, inan image space where decoded signals are provided to a helmet attitudecontrol and to a display control for providing actuator control signalsto at least a helmet attitude actuator mounted in a frame such as anarcade booth and to a helmet mounted display.

FIG. 8 shows one of the three attitude sensing planes of FIG. 7 forsensing pitch attitude of the cameraman's head, according to theinvention.

FIG. 9 shows a series of steps which may be carried out by the encoderof FIG. 7 prior to encoding the pitch control information for subsequentuse in the image space, according to the invention.

FIG. 10 shows a series of steps that may be carried out in the encoderin the object space for encoding the video, head attitude and eyedirection data, according to the invention.

FIG. 11 shows a series of steps that may be carried out in the decoderin the image space, according to the invention.

FIG. 12 shows more details of the relation between the sensors andencoder in the object space of FIG. 7, according to the invention.

FIG. 13 illustrates aspects of the sensors and encoder of the objectspace as well as the decoder and controls of the image space in moredetail, according to the invention.

FIG. 14 shows in an expanded view the details of the motion-controlledhelmet of the image space of FIG. 7, according to the present invention.

FIG. 15 shows a perspective view of a universal-joint such as may befixedly mounted within the joint 106 of FIG. 14.

FIG. 16 shows in plan view the U-joint of FIG. 15 within the gear ofFIG. 14.

FIG. 17 shows a coordinate system appropriate for the motion-controlledhelmet of FIGS. 7 and 14 by means of which the sensed signals in theobject space of FIGS. 7, 12, and 13 can be transformed into attitudinalmovements of the viewer's head.

FIG. 18 shows a series of monocular images gathered by a cameraman in anobject space for presentation to a passive viewer as nonuniform images,according to the invention.

FIG. 19 shows a series of stereo images similar to those of FIG. 18,according to the invention.

FIG. 20 shows a pair of eyes fixating at different points.

FIG. 21 shows a series of stereo image pairs similar to those of FIGS.18 and 19 that achieves high resolution binocular viewing, exceptwithout a need for eye tracking in the object space, according to theinvention.

FIG. 22 shows a moveable headrest for supporting the head of a user inexecuting head movements while viewing images from a changing direction,according to the invention.

FIG. 23 shows a user reclining on a support in the form of a reclinerequipped with a moveable headrest while viewing images from a changingdirection, according to the invention.

FIG. 24 shows a user standing on a support in the form of a humanactivity simulator equipped with a moveable headrest while viewingimages from a changing direction, according to the invention.

FIG. 25 shows a user seated on a support in the form of a chair equippedwith a moveable headrest while viewing images from a changing direction,according to the invention.

FIG. 26 shows a moveable headrest with a pivotal support assembly forsupporting the head of a user in executing head movements while viewingimages from a changing direction.

FIG. 27 shows the head of the user of FIG. 26 from above as the userviews a scene with a straight-ahead direction-of-view.

FIG. 28 shows the head of the user of FIG. 26 from above as the userviews a scene with a leftward direction-of-view.

FIG. 29 shows the head of the user of FIG. 26 from above as the userviews a scene with a rightward direction-of-view.

FIG. 30 shows an open-loop proportional control for a moveable headrestused passively.

FIG. 31 shows a closed loop proportional-plus-integral control for amoveable headrest used passively.

FIG. 32 shows hardware setup for active use of the moveable headrest,i.e., with the user moving his bead at will.

FIG. 33 is the same as FIG. 32 except that the reality engine 70 b isremote and is accessed via a network 74 b.

FIG. 34 shows passive use of the moveable headrest with an actuator 14 cmoving the headrest and hence the user's head in response to a signal ona line 16 c from a local reality engine 70 c.

FIG. 35 is the same as FIG. 34 except the reality engine is remote.

FIG. 36 shows a video camera collecting images of a scene illuminated bya light source with an eye of a cameraman shown using an eyepiece toview the scene being photographed.

FIG. 37 shows apparatus for showing images to an eye of a user withappropriate granularity.

FIG. 38 shows light rays projected to form an image that fills or almostfills the entire area or extent of a screen shown in FIG. 37.

FIG. 39 shows light rays projected to form an image that only partiallyfills the entire extent of the screen of FIG. 37.

FIG. 40 shows light rays projected to form an image that only fills asmall extent of the entire extent of the screen of FIG. 37.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

FIG. 1 shows an apparatus 10, according to the present invention, forproviding light 12 to an eye 14 in a head 16 of a passive viewer forformation of images in the eye 14. The apparatus 10 comprises a lightsource 18, responsive to a light control signal on a line 20, forproviding the light 12 for the formation of the images with a changingpoint of view. The light source 18 may be any kind of device forproviding light which is cast on the retina to form an image. In otherwords, it should be realized that the light source 18 could be any typeof light source, including but not limited to the many various knowntypes of displays. Such displays may include those where an image isfirst formed outside the eye, including but not limited to any type ofhead mounted display (HMD) or, e.g., any type of display where the imageis first formed inside the eye including any type of virtual retinaldisplay (VRD) See U.S. Pat. Nos. 5,184,231; 5,189,512; 5,406,415;5,293,271; 4,969,714; 4,968,123; and 4,961,626 for typical examples ofHMDs. See U.S. Pat. Nos. 5,467,104, 5,596,339 and 5,574,473 for examplesof VRDs.

A head guide 22 is connected to the light source 18 and is responsive toa head guide control signal on a line 24, for mechanically changing theorientation of the light source 18 in correspondence with the changingpoint of view for guiding the head 16 of the viewer in a correspondinglychanging orientation for viewing the images with the eye 14 in the head16 of the viewer at the changing orientation of the light source 18 andfrom the changing point of view. “Changing orientation” is used in thesame sense as one or more of the pitch, roll, or yaw components ofchanging attitudes for aircraft or spacecraft. The head guide is thus alight source actuator or mover of the light source in order to be aguider of corresponding movements of the head of the viewer.

A mechanical head coupler 26 for coupling the head 16 of the viewer tothe light source 18, or even to the head guide 22 as indicated by a line26 a may, but need not, be provided as well for assisting in guiding thehead of the viewer in changing its orientation in following the changingorientations of the light source. In other words, the function of thehead coupler 26 is to provide a way to couple the head or face of theviewer to the display so as to facilitate guidance of the head of theviewer 16 in passively following movements of the light source 18 ascontrolled by the head guide 22. Such a head coupler 26 can take theform of a hollow casing, a portion being concave to fit about the faceof the head of the viewer 16, said portion have two eye openings to belooked through by the viewer's eyes as shown, e.g., by M. L. Heilig inU.S. Pats. No. 2,955,156 or 3,050,870 or as similarly shown, morerecently, in a wrap-around viewer by the Nintendo “Virtual Boy” (Part #32787 Item # NESM128). It could simply be eyecups which the viewerinserts his eyes into and rests his brows, cheeks or eye-orbits against.It could be a head or face “rest” that is rigidly attached to the lightsource as shown in FIG. 3 of U.S. Pat. No. 5,584,696 where the viewerlooks through a single aperture. It could be a helmet or headrest. Inother words, it can be any means for more or less weakly or stronglymechanically coupling the head of the viewer to facilitate the functionof the head guide in mechanically guiding the head of the viewer tofollow the movements of the light source.

With the passive viewer seated or standing before the light source, thehead 16 of the viewer is coupled to the light source 18 by the means forcoupling 26. The light source provides light 12 to the eye for formingan image in the eye from a particular point of view as controlled by thelight control signal on the line 20. As long as the point of viewremains the same, the head guide 22 does not change the orientation ofthe light source and the scene remains the same. However, the lightcontrol signal can then gradually or even quickly change the point ofview and the head guide will at the same time correspondingly change theorientation of the light source according to the dictates of the headguide control signal. The head of the viewer is thus guided in such away as to feel the changing orientation of the point of view as ifexperiencing the change in reality.

For a simple case such as shown for example in FIG. 2A, the means forchanging the orientation of the light source (in the form of a displayhousing 18 a) may be supported on a fixed support 28 on a base 29 sothat the position of the display housing, to which it is connected,e.g., by means of a one-or-more axis wrist, e.g., a three-axis wrist 22a, is also essentially fixed, ie., with respect to translations inposition. The base shown is a ceiling or roof base but could as well bereversed as a floor base. As shown in FIG. 2B, the base could be anysurface such as a desk 29 a with a support 28 a, 28 b holding thedisplay much like a desktop computer except with a wrist 22 b. Both ofthe wrists of FIGS. 2A & 2B have an actuator such as a motor in one ormore axes for causing movement thereabout. The motor can be of any typeincluding electrical, hydraulic, or the like. The display housings 18 a,18 b are shown with the head coupler in the form of simple eyecups 26 a,26 b such as found on binoculars. Behind each eyecup may be a separatelight source such as shown, e.g., in U.S. Pat. No. 4,406,532. Otherkinds of displays without separate eyecups are equally usable. Handgrips30 may be provided as part of such means to assist the viewer incoupling his or her head to the eyecups. This embodiment is somewhatsimilar to a submarine periscope except that such a periscope has atelescoping support for changing the vertical position of the scope andonly has a single degree of freedom of orientation, i.e., for changingthe orientation of the scope about the axis of the support.

On the other hand, the head guide 22 of FIG. 1 may, in addition to beingcapable of changing the orientation of the light source, be equipped tochange the translatory position of the light source. As shown in FIG. 3,the means for changing the orientation of a light source 18 c, e.g. inthe form of a motorized wrist 22 c, can be mounted on a means 32 forchanging not only the orientation but the position of the display aswell. Such a means 32 can take the form of a selected robotconfiguration as shown, for example, in FIG. 3. Such a robot has a workenvelope having a shape depending on the selected configuration. Variousconfigurations are shown at pages 2154-62 of The Electrical EngineeringHandbook, CRC Press 1993, edited by Richard C. Dorf, in section 94.1entitled “Robot Configuration” at pages 2154-62 by Ty A. Lasky, Tien C.Hsia, R. Lal Tummala, and Nicholas G. Odrey, e.g., “Cartesian” asdescribed and shown at pages 2155-6, “cylindrical” as shown at page2156, “spherical” as shown at pages 2157, “articulated” as shown atpages 2157-8, “SCARA” as shown at pages 2158-9, and “gantry” as shown atpages 2159-61. As explained there, “configuration” refers to the way themanipulator links are connected at each joint. Each link will beconnected to the subsequent link by either a linear (sliding orprismatic) joint, or a revolute (or rotary) joint.

For example, as shown in FIG. 4, the means for changing the orientationof a light source 18 d, e.g., in the form of a wrist 22 d, can bemounted by a rigid bar 33 onto a cylindrical robot configuration 32 cfor changing the position of the display. It has one vertical revolutejoint 34 and two orthogonal linear joints 36, 38. Such a cylindricalrobot has a work envelope in the form of an annulus or part thereof asillustrated in FIG. 94.4 at page 2157 of The Electrical EngineeringHandbook, referred to above. Cylindrical robots are made by manydifferent manufacturers, e.g., the RT3200 or RT 3300 of SeikoInstruments USA, Inc., 2990 W. Lomita Blvd., Torrance, Calif. 90505.

There are many possible alternatives for creating the light controlsignal on the line 20 and the head guide control signal on the line 24of FIG. 1. For example, FIG. 5 shows a studio cameraman 40 havingheadgear with stereo cameras 44, 46 attached thereto for providing thelight control signal on the line 20 of FIG. 1. The cameras can be ofhigh definition, e.g., according to the new HDTV standard recentlypromulgated by the FCC or can instead be any other type for providing awide field of view image. See, for example, U.S. Pat. No. 4,323,925 foran image sensor suitable for resolving a high-resolution image. Thecameras can instead be mounted on rails 48, 50 of a structure 52 or inany other convenient location that permits the cameras to be mounted soas to be pointing in the same direction as the cameraman's head and soas to gather images of a scene from the moving point of view of thecameraman. The structure 52 is in turn connected to a counterbalancedarticulated arm 54 such as heretofore used for a different purpose inthe BOOM3C of Fakespace Inc., 241 Polaris Ave., Mountain View, Calif.94043. For example, it can have a first arm 56 connected at one end tothe structure 52 by joints 58, 60 and at the other end to acounterweight 62. A joint 64 connects the first arm 56 to a second arm62 which is in turn connected to a pedestal 64 by additional joints 66,68. The joints need not be located at the exact same positions shown butcan be changed, if a different design is desired. Each of the joints hasa transducer associated therewith for monitoring rotation about the axisof the associated joint. Signal processing of signals from thetransducers provides an indication of position and orientation of thecameraman's head in space. Thus, the structure 52 is used for providingfull six-degree of freedom motion monitoring for providing the movercontrol signal on the line 24 of FIG. 1. The articulated arm 54 ismounted on a pedestal 56 which may be stationary or movable on a studiodolly. For the embodiment shown, the cameraman can move his head indifferent orientations and positions. He can walk about within theconfines of the work envelope of the articulated arm and thus providethe light control signal 20 of FIG. 1 for use by the light source inproviding light at a point of view that changes with movements of thecameraman's head. It should be realized that there are numerous othersimilar mechanical monitors available. See, e.g., U.S. Pat. No.4,586,515 for a device for measuring the position and/or motion of abody part.

As mentioned above, it should be realized that there are alternativeways to create the light control signal on the line 20 as well as themover control signal on the line 24. FIG. 6 shows a generalized means 70for indicting head attitude, as well as head position if desired. Themeans 70 can take the form of a mechanical head-coupled tracker 72 suchas shown in FIG. 5 for providing a mover control signal on a line 24 a.An OR gate 74 is shown in the means 70 of FIG. 6 merely to indicate thatthere are alternatives to the arrangement of FIG. 5 for providing amover control signal on a line 24 c to a signal storage medium 76. Suchan OR gate 74 would not actually be present since only one alternativeis needed. It is shown for the purpose of indicating that there arevarious alternatives. Another way for indicating head position andattitude is to use a head mounted navigation system 78 for sensingposition and attitude with gyros, accelerometers, radio, magneticsensors, or the like for providing a mover control signal on a line 24b, such a system is shown in the object space of FIG. 7 below. Anotherapproach would be a system from Polhemus Inc., such as the systemmentioned above. Yet another approach is shown by a block 80 in FIG. 6which represents a way for a designer to provide a head guide controlsignal on a line 24 d. This could be done by a computer workstation.

FIG. 6 also shows a means 82 for providing a light control signal 20 ato the storage medium 76. It can comprise cameras such as head coupledcameras 84 for providing a light control signal on a line 20 b such asshown in FIG. 5 and in FIG. 7 below or it can comprise a means 86 suchas a computer workstation for providing a light control signal on asline 20 c. The two alternative signals on the lines 20 b, 20 c are shownprovided to a fictitious OR gate 88 to signify that either alternative84, 86 can be used. The OR gate 88 of course need not be actually andwould not usually be present. This is not to say, however, thatpost-production work could not be done on camera generated imagery bymeans of a workstation.

The signal storage medium 76 is responsive to the head guide controlsignal on the line 24 c and the light control signal on the line 20 afor storing them in timed relation to each other in such a way that theycan later be retrieved in the same timed relation and provided on lines20 d, 20 e as a combined output signal for use by a light source such asshown in FIG. 1. It should be understood that although the varioussignals on the lines 20, 24 may be shown as single lines that they caneach typically comprise a plurality of signal lines.

FIG. 7 shows in an object space 100 one of the particular means 78 forthe means 70 as well as one of the particular means 84 for the means 82of FIG. 6. It should be realized, however, that the various particularmeans of the means 70 of FIG. 6 can be used in various combinations withthe various particular means of the means 82 of FIG. 6.

FIG. 7 shows both the object space 100 and an image space 102, each ofwhich may have respective helmets 103, 104 therein, or theirequivalents, according to the present invention. The image space is forpresentation of images to a “viewer” while the object space is for imageacquisition, e.g., by an “observer.” The linguistic convention ofcalling the active creator of the image the “observer” in the “object”space and the passive consumer of the created image in the “image” spaceas the “viewer” will be used throughout.

According to the present invention, the helmet 104 in the image space102 is worn by a passive viewer (not shown), e.g., seated within astationary arcade-like housing (not shown) with respect to which thehelmet 104 is made to move. It is made to execute at least some minimummovement such as one or more attitudinal movements emulative of pitch,roll and yaw movements of the helmet 103 in the object space 100 andworn by a cameraman observer (not shown) who is free to move about whiledirecting his head and gaze in various directions. In other words,translatory movements of the head of the observer with respect to somereferent may, but need not be, emulated by the helmet of the viewer. Forthe detailed embodiment shown below, however, only the attitude of thecameraman's head is emulated in the image space. Translations areignored. This makes it possible for the embodiment shown for theviewer's head to remain relatively stationary (not be forced to undergovarious translatory accelerations) in the image space in a positionalsense. Such can be introduced but are best imparted to the body of theviewer rather than directly to the head. Such requires additional setsof external platforms and associated superstructures depending on thenumber of axes of control and are shown below in other embodiments. Forthe case shown where only head attitude is emulated, the viewer can beseated or standing in one position within a stationary structure.

It will be realized, therefore, that the invention can be used inplatforms in which the viewer can be controlled in other ways. Forinstance, the viewer could be seated as shown in FIG. 1 of U.S. Pat. No.5,515,078 with the display FIG. 1 of that patent controlled in themanner disclosed herein. The display 21 of that patent could bepositioned as shown therein or more closely, with its arm 11 angled tobring the display 21 close to the face of the user to facilitate use ofa head coupler as taught herein. In addition to controlling theorientation of the display 21 as taught herein, however, the positionand/or orientation of the seated viewer can be controlled as shown inFIG. 3 of U.S. Pat. No. 5,515,078 except not based on the joystick 30,35 choices of the viewer, but passively, based on monitoring of theposition and/or orientation of the cameraman in the object space basedon the same principles as disclosed herein. The invention can similarlybe used with like devices such as shown in U.S. Pat. Nos. 5,388,991;5,551,920; 5,453,011; and 5,353,242, among many others.

At the same time, a helmet mounted display 106, to be described below,mounted on the helmet 104, provides images to the passive viewer wearingthe helmet 104 that are gathered by the cameraman “observer” in theobject space 100 wearing the helmet 103 with cameras mounted thereon.The images viewed by the passive viewer in the image space are thereforepresented in such a way as to be emulative of images seen by thecameraman as he moves his head, at least attitudinally, in the objectspace. It should be realized that translational position of theobserver's head can be monitored in the object space as well, withrespect to a selected referent, and such translations can be emulated bythe helmet 104 in the image space by means of the already mentionedadditional platforms and associated superstructures (not shown). Or,rather than urging the head of the viewer to translate directly, suchpositional translations can instead be imparted to a platform supportingthe body of the viewer, as shown below in other embodiments. It shouldbe mentioned that the body of the cameraman can be monitored as well orinstead of the head, at least for purposes of translation.

The helmet 103 in the object space 100 has at least one camera 108, andpreferably a second camera 110 as well, mounted on opposite sides of thehelmet 103 for respectively gathering monocular or preferablystereoscopic images of objects in the object space according to thecameraman's head movements. The cameras provide image signals on lines112, 114 to a signal processor or encoder 116 where they are encoded fortransmission to the image space 102 via a signal on a line 118.

Also illustrated mounted on the helmet 103 are helmet attitude sensors120, 122, 124 such as, but not limited to, accelerometers for monitoringthe cameraman's head attitude, respectively, its pitch (P), roll (R) andyaw (Y). Opposite on the helmet to each of the illustrated sensors 120,122, 124 may be located corresponding sensors 126, 128, 129 (not shownin FIG. 1) as twins to sensors 120, 122, 124 for sensing equidistantlyon opposite side of the helmet. For example, as shown in FIG. 8, thepitch (P) sensor 120 on the front of the helmet 103, e.g., just abovethe visor, may have a corresponding twin pitch sensor 126 (not shown inFIG. 7) on the back of the helmet opposite to the sensor 120. These twosensors are shown in FIG. 8 located on positive and negative equallyspaced sides of a y-axis in a y-z plane of a three-axis (x,y,z)coordinate system (having three such mutually orthogonal planes) havingan origin centered on the head of the observer at a point 130. Such athree axis system is illustrated with the origin 130 translated to apoint 130 a for purposes of clarity, above the helmet 103 of FIG. 7 butit should be understood that it is most convenient to position theorigin of the illustrated coordinate system at the point 130 at thecenter of the cameraman's head, as shown. Of course, the origin can belocated at any convenient point and translated as desired by appropriatecoordinate translations and transformations.

In any event, the two sensed pitch signals from the accelerometers 120,126 of FIG. 8 may be used together to be indicative of pitch (P)rotations in the y-z plane about the point 130 midway between them,e.g., in the center of the cameraman's head. It should be realized,however, that a single sensor can suffice. Similarly, twin roll and yawsensors may be positioned at equal distances apart (in correspondingmutually orthogonal roll and yaw sensing planes) on opposite sides ofthe helmet for sensing roll and yaw motions about substantially the samecenter point 130. For example, as shown in the object space of FIG. 7,the roll acceleration sensor 122 may be positioned as shown on thehelmet over left ear of the observer and oriented as shown on a positiveside of the x-axis while a not shown acceleration sensor 128 may besimilarly positioned over the right ear of the observer on the otherside of the helmet on a negative side of the x-axis. Together, they maybe used inter alia to measure rotations in the x-z axis about the point130 in the center of the cameraman's head. Similarly, the accelerationsensor 124 of FIG. 7 may be positioned over the left ear of the observerand oriented as shown in FIG. 7 on the positive x-axis with a not shownacceleration sensor 129 similarly positioned over the right ear of theobserver on the other side of the helmet on the negative x-axis.Together, they may be used inter alia to measure rotations in the x-yaxis about the point 130. It should be realized that it is also possibleto monitor the attitude of the cameraman's head with any appropriatesensor with respect to another referent, such as but not limited to hisbody.

The sensors need not be accelerometers but could be gyros of theelectromechanical type, SAGNAC effect fiber optic gyros, or conceivablyeven more bulky laser gyros. Other types of attitude sensors based onmagnetic sensors, light beam sensors, radio sensors, or the like, areknown and are of course useable as well, as will be understood by thoseof skill in the art.

It should be realized that although FIG. 7 shows the image creationprocess as taking place in an “object” space 100 by means of a camera ona helmet gathering images of real objects and mounted on a helmet,similar images can be created in other ways, e.g., by animation orsuccessive images created on computers, using software, as suggestedabove in connection with the block 80 of FIG. 6.

In any event, as shown in FIG. 7, the sensed attitude signals mayaltogether be provided on an illustrative line 131 to the signalprocessor or encoder 116 for being encoded along with the image signalson the lines 112, 114. They may be combined, for example, using timedivision multiplexing techniques or by any other convenient technique.Or the signal processor or encoder can calculate the attitude of thecameraman's head based on the sensed signals in the object space andencode appropriate actuator signals for transmission to the image space.It should be realized, however, that the helmet attitude and imagesignals need not be processed and combined into a single signal on theline 118, but may be processed and provided separately. Each of theacceleration signals may be separately processed in the signal processoror encoder 116 to provide an indication of angular displacement in eachof the separate pitch, roll and yaw axes. For example, the pitch axismay, but need not, be defined in the y-z plane shown by the x-y-z axesof FIG. 7 centered at the point 130 in the object space of FIG. 7.Similarly, the yaw axis may, but need not, be defined in the x-y planeand the roll axis in the x-z plane.

FIG. 9 shows a signal processing method which may be used for evaluatingthe sensed acceleration signals to determine the cameraman's headattitude in the y-z (pitch) plane of FIGS. 7 & 88. Though not shown, asimilar signal processing method may be used for evaluating the sensedacceleration signals in the x-z (roll) and x-y (yaw) planes. At theoutset, it should be realized that other equally effective coordinatesystems (such as polar or cylindrical coordinate systems) and methodsmay be used and the following is just one example.

According to FIG. 9, after entering in a step 131 a, a pair ofinitialization steps 132, 134 are executed to set a rotation variableRTN(R) and a translation variable XLTN(T) equal to zero at a selectedcameraman reference attitude and position, e.g., standing erect and headpointing straight-ahead with respect to a selected axis. For thisexample, the variable XLTN(T) represents the position of the point 130with respect to the z axis of FIG. 7 or 8.

After initialization, a decision step 136 is executed to determine ifacceleration (A) has been sensed by the accelerometers 120, 126 of FIG.8. If not, then the step 136 is re-executed until such is sensed. Due totheir bidirectionality and orientation in the z direction, both sensorswill sense an acceleration along the z axis whether it be positive ornegative. Once acceleration is sensed, a step 138 is executed todetermine if a rotational acceleration is sensed by the accelerometersor not. I.e., if the accelerometers sense translations in oppositedirections at the same time, this is interpreted as sensing a rotation.

If a rotation has been sensed, a decision step 140 is executed todetermine if the sensed rotation is a pure rotation in the y-z planeabout the point 130 or if it is accompanied by a translation of thepoint 130 in the z direction. It can do this by comparing the absolutemagnitudes of the oppositely sensed accelerations of the two sensors120, 126. If they are equal, then they represent a pure rotation. Ifnot, then there is also a translation present in the positive ornegative direction.

In the case where they are not equal, a step 142 may be executed todetermine the magnitude of the equal and opposite sensed accelerationsthat are together indicative of the magnitude of the arc of rotation andits direction. The equal but opposite accelerations will either bothindicate a clockwise rotation or a counterclockwise rotation. Forinstance, if sensor 120 indicates an acceleration in the positive zdirection and sensor 126 indicates an acceleration in the negative zdirection then the rotation is in the clockwise direction.

If knowledge of translations is desired, the step 142 can be executed todetermine the magnitudes of the two oppositely sensed accelerations andthen, in a step 144, to determine the part of one of the sensedaccelerations that exceeds the other, i.e., to determine the difference(ΔA) between the sensed accelerations. The difference can then beintegrated twice to determine the length of the translation and summedwith the previous value of XLTN(T), as indicated in a step 146, toindicate the current z position. Such knowledge may be needed forinstance in all three orthogonal planes, not just the y-z plane, whereit is desired to keep track of the three dimensional translatoryposition of the head of the cameraman. Such is not used in the presentapplication but it could be used in other applications.

In a step 148, executed subsequent to either the translation update step146 or the decision step 140, the sensed acceleration (A) (i.e., theacceleration that is equal in terms of absolute value in both sensors)is twice integrated to determine the length of arc of rotation of themovement of the cameraman's bead about the point 130. The doublyintegrated acceleration is summed in a step 150 with the previous valueof the pitch rotation value RTN(R). Since the radius (r) from the point130 to each of the accelerometers is known and since the arc of rotationis known from the foregoing, the angle of rotation in the y-z plane canbe determined. I.e., if the radius (r) of a circle is known, the lengthof an arc (a=RTN(R)) on the circumference can be used to measure thecorresponding angle (P) at the center. Consequently, the pitch angle (P)can be determined in degrees as shown in a step 152 according to therelation 2ππ/RTN(R)=360/P. At this point, as described in more detailbelow, calculation of the necessary movement of the platform 153 of FIG.14 in the y direction can be made and output for encoding with the videosignal as indicated in steps 154, 156.

The step 136 is then re-executed to determine if additionalaccelerations have been sensed. Once again, if no additionalaccelerations have been sensed then the step 136 is executed again adinfinitum until an acceleration is sensed. At that time, the decisionstep 138 is executed to determine if there has been a rotation.

If it is important to determine translations in the y-z plane, a step158 may be executed after the step 138 to determine the magnitude anddirection of the acceleration (A) from the sensors 120, 126. The step160 is next executed in order to twice integrate the sensed acceleration(A). This determines a pure translation which is added in a step 162 tothe previous value of the translation variable XLTN(T). A return is thenmade to the step 136 to determine if additional accelerations have beensensed. Once again, if no additional accelerations have been sensed thenthe step 136 is executed again ad infinitum until an acceleration issensed. At that time, the decision step 138 is executed to determine ifthere has been a rotation.

As mentioned, a similar signal processing method as shown in FIG. 9 maybe executed at the same time (serially or in parallel) for evaluatingthe sensed acceleration signals in the x-z (roll) and x-y (yaw) planes.These various signal processing procedures may be carried out as shownin FIG. 9 by means of the signal processor or encoder 116 of FIG. 7shown in detail in FIG. 12. As also mentioned, for the embodiment shown,translations need not be tracked. Here, it is desired to “divorce”attitudinal motions of the cameraman's head from translations thereofbecause of the stationary nature of a structure 164 with respect towhich the body of the viewer in the image space 102 is also relativelystationary. In other words, for the preferred embodiment, it is notdesired to positively translate the head or body of the viewer in theimage space. So, for the illustrated embodiment, the steps 144, 146,158, 160, 162 may be omitted. I.e., if the answer to the question posedin the decision step 138 is negative, then the step 136 may bere-executed directly and steps 1158, 160, 162 may be omitted completelyfrom the procedure.

FIG. 12 shows the signal processor or encoder 116 of FIG. 7 as a generalpurpose signal processor capable of carrying out the steps of FIG. 9. Itmay include an input/output (I/O) device 168 which may be represent dinpart by a device 168 a for interfacing with an attitude monitor 170which may, but need not, include the accelerometers 120, 122, 124, 126,128, 129, a left eye monitor 169 a, and a right eye monitor 169 b. Thesemay be any kind of eye monitor such as for monitoring accommodation,position, direction, or the like. It may also include various data,address and control busses 172 for interfacing with a central processingunit (CPU) 174, one or more memory devices which may include aread-only-memory 176 and a random access memory (RAM) 178. The I/Odevice 168 may also be represented in part by a device 168 b forinterfacing with the image space 102 over the line 118.

The left and right eye monitors 169 a, 169 b of FIG. 12 may take theform of a pair of left and right oculometers 180 a, 180 b. Such a pairof oculometers are capable of monitoring the eyes' attitudes ordirections (sometimes called “positions”) and providing a monitoredsignal such as shown on a line 133 which represents signals indicativethereof. An oculometer device 180 of the illustrated eye directionsensor, for one or both eyes, can for example take the form of aninfrared source for illuminating the cameraman's eye which reflects theinfrared light into a directionally sensitive infrared sensor inside thehelmet (not shown) for sensing the direction of eye movement. Such isshown for example in U.S. Pat. No. 4,034,401, among many others. Itshould also be realized that other types of eye monitoring methods maybe used as well, and further that it is not necessary to monitor eyeattitude at all. Eye attitude is chiefly useful, according to thepresent invention, for providing a control signal for controlling anarea of image nonuniformity such as an area of high resolution, dynamicrange, brightness, or the like, in the images displayed to the passiveviewer. It many not be necessary in some applications to have such acontrol signal (and consequently no eye monitoring) since uniform imagescan certainly be used. Equally, it should be realized that eye attitude,if monitored, can be monitored in any number of axes of eye attitude,including not only pitch and yaw but roll, i.e., torsion (rotation aboutthe visual axis), and could even include, beyond attitude, positionmonitoring (in the strict sense of the word, i.e., small translations ofthe eyeball within its socket), although torsion and position monitoringare currently deemed not necessary for a preferred embodiment.

All of the various sensed signals in the object space 100 may berepresented as a generalized group of sensed signals on the bundle oflines 132 (see FIGS. 7 & 12) for being encoded in a convenient format inthe signal processor or encoder 116 for being provided on the line 118to the image space 102. Encoding can take place in the encoder 116, forexample, as shown in FIGS. 10 & 12. After entering in a step 184, ablock of a selected size of video data from the cameras is input andstored in a step 186. A step 188 is next executed to input and store eyedirection data from the oculometer sensors 180 a, 180 b. Head attitudedata is next input as indicated in a step 190. Once the sensed headattitude, eye direction and video data is input and stored, a step 192is then executed in order to retrieve the eye direction data stored inthe step 188. Similarly, at least a part of the video data previouslystored in the step 186 is retrieved in a step 194. The retrieved videodata is then encoded in a step 196 according to the retrieved eyedirection data. That is, if the eye direction signal indicates that thevideo data to be encoded is in a portion of the overall image that is tobe encoded with a higher or lesser degree of resolution, then theencoding is carried out accordingly. A decision step 198 is thenexecuted to determine if the encoding of the video block is done. Ifnot, the steps 192, 194, 196 are again executed until the block isencoded.

It should be realized that the eye attitude signal can instead be usedto directly control the attitude of a 3-axis platform mounted e.g. onthe helmet (or elsewhere) and having a camera with a nonuniform lens(e.g., U.S. Pat. No. 3,953,111) mounted thereon which is therebydirected in the same direction as the cameraman's eye. In that case, theoptics of the camera effects the desired nonuniform imagery andnonuniform encoding techniques are not needed. Similar 3-axis cameramounts could be used for the cameras of FIG. 5.

Head attitude data stored previously in the step 190 is next retrievedas indicated in a step 200. The head attitude data is then encoded withat least part of the selected video block as indicated in a step 202. Adecision step 204 then determines if the encoding is done. If not, thesteps 200, 202 are repeated until it is determined in the step 204 thatthe block is encoded. If a variable focal length device such as a device205 of FIG. 7 is used, a step 206 is executed to retrieve the eye datainput and stored in step 188. This may be eye direction or similar data.A step 208 is next executed to determine the distance from the monitoredeyes to the point of fixation converged upon by the monitored eyes. Thisinformation is then encoded in the selected block of data as indicatedin a step 210. After that, as indicated in a step 212, the encoded blockis stored or transmitted directly to the image space and the step 186 etseq. is executed again.

It should be understood that numerous alternative encoding techniquescould be carried out as well including analog techniques using dedicatedanalog circuitry. Anyone of skill in the art could devise a signalencoding technique for transmitting both the video and controlinformation required in the image space based on the teachings hereof.For instance, the video image may be encoded in the conventional analogmanner with odd and even raster fields which are interlaced to form asingle frame. In that case, several of the horizontal video lines at thetop or bottom of a field can be used for encoding the changing pitch,roll and yaw control information. For a given horizontal line used for acontrol purpose, a selected fixed voltage level between “white” and“black” levels will indicate the delta x, delta y or yaw rotationdescribed in FIGS. 14-16 and 17 below. Such a control concept is shownfor example in FIGS. 2-3 of U.S. Pat. No. 4,513,317 albeit for adifferent purpose. It should be realized that the encoding technique isnot restricted to conventional digital techniques but could take otherforms such as, but not limited to, a new HDTV format. Other as yetundefined formats such as for virtual retinal displays may also be used.It should also be realized that the signal on the line 118 need not beprovided directly to the image space 102 but can instead be stored on amechanical, magnetic, optical, electronic, or the like storage medium214 for subsequent transfer as indicated on a line 216 for playback on aplayback device 218 in the image space. The bundle 132 can take the formof a wire harness connected to the encoder 116 which may be carried bythe cameraman in a backpack, for example, along with the storage device214.

A signal processor or decoder 220 in the image space 102 of FIGS. 7 & 13is responsive to the encoded signal on the line 118 either directly(live) from the object space or prerecorded and played back on theplayback device 218 and provided on a line 219. The decoder providesdecoded image signals on a line 222 to a display control 224 whichprovides a signal on a line 226 for controlling the display 106 whichmay be mounted on the helmet 104 which may be monoscopic orstereoscopic, as described previously. The decoder 220 also provides adecoded helmet attitude signal on a line 226 to a helmet attitudecontrol 228 which in turn provides a helmet attitude control signal on aline 230 to a plurality of actuators such as three actuators 232 a, 232b, 232 c mounted on a stationary plate 234 for actuating the helmet 104in a corresponding plurality of axes such as three axes, as shown,emulative of the motions of the helmet 103 in the pitch, roll and yawaxes sensed in the object space 100. The attitude control 228 may, e.g.,be a simple open loop having proportional plus integral gain. Althoughnot shown, sensors could be provided on the platform 234 to senseposition of the plate, for example, for providing feedback signals for aclosed loop control. In any event, the control 228 provides actuatorcommand signals on the line 230 for causing the actuators to carry outthe control strategy described in connection with FIG. 17 below.

The decoder 220 may also provide a variable magnification control signalon a line 236 to the variable magnification device control 205 which inturn provides a variable magnification device control signal on a line238 to a variable magnification device 239 (see FIG. 13) associated withthe display 106 and as disclosed in more detail in copendingapplications having Ser. Nos. 08/025,975 and 08/001,736, now U.S. Pat.No. 5,422,653. Of course it should be realized that one or more or evenall of the signal processing for the control functions carried out inthe image space by the controls 224, 205, 228 need not be carried out inthe image space but could equivalently be carried out in the objectspace based on raw data available on the line 132 in the object space.

The plate 234 may, but need not, be fixedly mounted on a structure (notshown) that, e.g., has fixed structural parts 164 that are indicative ofstructurally stable parts of a mount for the plate 234 such as anarcade-like booth within which a viewer may be standing or seated forplacing the helmet 104 on the viewer's head for guiding or inducing headmovements of the viewer for passive viewing of images gathered in theobject space by the cameraman in the helmet 103. In other words, theseated or standing passive viewer wearing the helmet 104 in the imagespace 102 is induced or guided to at least emulate pitch, roll and yawhead movements corresponding to the corresponding monitored headmovements of the cameraman in the object space, in response to theencoded attitude information in the signal on the line 118, while at thesame time watching corresponding images gathered by the cameras 108, 110also encoded on the line 118 and decoded in the image space for passiveviewing.

FIG. 13 shows in more detail the image acquisition apparatus in theobject space 100 of FIG. 7, according to the present invention. Theobject space 100 of FIG. 13 shows a cameraman's head 103 a which may beinserted in the helmet 103 of FIG. 7 and may therefore be considered tohave a common coordinate system origin. The cameraman's head isillustrated as having a pair of left and right eyes 14 a, 14 b that aremonitored by left and right eye monitors 169 a, 169 b that providesensed eye attitude signals on lines 240 a, 240 b to respective left andright camera controls 242, 244. These in turn provide camera controlsignals on lines 246, 248 to the left and right cameras 108, 110,respectively. As mentioned, according to the invention, these controlsignals may, but need not, be used to control the relative position of aregion of image nonuniformity including but not limited to a region ofincreased resolution, dynamic range, or the like, within each of theimages provided to the passive viewer in the image space. Such a regionis emulative of the increased sensitivity of the fovea of the particularmonitored eye along the visual axis thereof.

The image space 102 of FIG. 13 is the same as shown in FIG. 7 exceptalso showing a variable magnification device 239 which may be used withthe display 106 of FIG. 7 so as to provide images with variablemagnification, i.e., at various apparent distances. In other words, thedevice 239 causes the images from the display 106 to be provided in sucha way as to cause the eyes of the viewer to accommodate differently forthe various successive images presented thereto. The device 239 may bethe same or similar to that shown in copending U.S. patent (applicationSer. No. 08/025,975) or in the copending U.S. patent application Ser.No. 08/001,736, now U.S. Pat. No. 5,422,653, particularly in connectionwith FIGS. 3, 5, 6, 17, 19, 20, 21, and 26-35 thereof.

FIG. 11 shows a process that may be carried out in the decoder 220 fordecoding the signal on the line 118. After entering in a step 250, astep 252 is executed to extract the information encoded on the line 118.The video information may be decoded first, as indicated in a step 254.The video signal on the line 222 is then provided from the decoder 220to the display control 224. A step 258 may next he executed to decodethe helmet attitude data. In a step 260, the helmet attitude controlsignal on the line 226 is provided. After that, the previously encodeddistance information is decoded in a step 262, and the variablemagnification device control signal on the line 236 is provided in astep 264. A return is then made in a step 266.

FIG. 14 illustrates the example of a motion-controlled helmet 104 ofFIG. 7 in enlarged detail, according to the present invention. Astructure (not shown) such as a stationary arcade-type booth or a movingpositional and/or attitudinal simulator such as a vehicle simulator, hasthe platform 234 mounted fixedly within. This embodiment includes amoveable platform 153 mounted on the inside of the not shown structure,e.g., in the inside top part of a booth structure as an extensionthereof. The booth may be designed for having the passive viewerstanding or seated. Several pedestals 268, 270, 272, 274 are mountedfixedly on the platform 234. The moveable platform or plate 153 iscontrolled in the x-y plane by a pair of orthogonal, screw gear drives232 a, 232 c corresponding to two actuators of the three actuatorembodiment of FIG. 1. The first screw gear drive 232 a includes amotor-gear assembly that drives the plate 153 by means of a screw 276 inthe plus or minus x-direction. Similarly, the motor-gear assembly 232 cdrives the plate 153 by means of a screw 278 in the plus or minusy-direction. Mounted perpendicularly at the ends of the screws 276, 278are slide bars, such as the slide bar 280 shown at the end of the screw278. The slide bar 280, e.g., is slidably mounted within a slide guide282 and the screw 278 is attached to a point of the slide bar but canrotate on that point. Similarly, stabilizing rods 284, 286 can beinstalled on opposite sides of the plate 153 with similar slides inslide guides in axial alignment with corresponding screws 276, 278 inorder to give the control a framework within which to push the plate 153about in the x-y plane.

The third actuator 232 b turns a gear 288 that turns another gear 290that has a universal-joint such as shown in FIG. 15 that has an internalpart 292 (see FIG. 15) with a square hole 294 within that accepts asquare rod 296 fixed on the helmet 104 for connection thereto, e.g., byslidable insertion therein. The internal part 292 is connected by pins298, 300 (see FIG. 16) to an intermediate part 302 along the x-axis sothat the parts 292 and 302 are freely rotatable with respect to eachother along the x-axis. The intermediate part 302 is in turn connectedto an outer part 304 that has an outside square boundary 306 that fitstightly in a square hole in the gear 290. The intermediate part 302 isconnected to the outer part 304 by pins 308, 310 along the y-axis sothat the parts 302, 304 are freely rotatable with respect to each otherabout the y-axis.

It will be realized that the illustration of FIG. 14 is for teachingpurposes and the motor 232 b will have to be fixedly attached in someway, e.g., by a bracket (not shown), to the plate 153. The gears 288,290 are likewise rotatable within casings (not shown) fixed to the plate153. The square rod 296 is connected to the U-joint of FIG. 15 andslides up and down freely through the square hole 294 of the universaljoint. Similarly, the rod 296 of the helmet 104 is not shown actuallyinserted in the universal joint for purposes of clarity. It will also berealized that the universal joint need not take the form shown, sincemany other u-joints are known, and that even if the form taken isgenerally the same, the various parts of the universal joint need not besquare. The display 106 is shown mounted on the helmet but the displaymay instead be a panoramic display mounted in a stationary manner withrespect to the not shown structure 164. The three actuators 232 a, 232b, 232 c are separately responsive to corresponding separate components230 a, 230 b, 230 c of the control signal 230 of FIG. 7 for beingactuated thereby.

It should be realized that the number of axes of control need not be asextensive or could even by more extensive than that shown, since manysimplifications or elaborations are quite possible. It was alreadyindicated above that it was not desired, for the preferred embodiment,to control position (translations) per se. It was preferred to leave thecontrol of head positioning to the viewer himself given that his head'spitch, roll and yaw axes were being so fully constrained. This freedomis granted to the viewer by making the rod 296 freely slidable withinthe U-joint. Position was therefore divorced from attitude in thepreferred embodiment and only attitude was positively controlled. Inthis way, the viewer could have the freedom to move his head along theaxis of the rod 296. It should be realized, however, that it would bepossible to more positively control position, at least to some degree,i.e., with respect to the fixed referent, such as the arcade booth, bysimply adding another actuator to the device of FIG. 14 for retractingor extending the rod 296 (e.g., with teeth added) in or from the U-jointand making some minor modifications thereto. In other words, it shouldbe realized that there are many different ways of connecting or couplingan actuator to a passive viewer's head for controlling or guiding themovements thereof and the invention is broadly directed to having anapparatus that can be controlled to move a display to guide the passiveviewer's head to allow the viewer to view an image in a manner emulativeof a corresponding active viewer's head movements. It should be realizedthat the sensed attitude signals of FIG. 7 need not be actually sensedbut can instead be dictated by a pre-planned program of head movementsby a workstation 80. It will therefore be understood that the variousdevices including actuators shown here are merely illustrative of theinvention and many other embodiments are within the scope of the claims.

With this in mind, an example will now be given of how to use the sensedhead attitude signals from the image space to cause the actuators 232 a,232 b, 232 c to actuate the helmet, using the exemplary actuator ofFIGS. 7 and 14, in such a way as to cause the passive viewer to emulatethe corresponding attitudinal head movements of the cameraman. FIG. 17shows such an example where the platform 234 of FIGS. 7 & 14 is shownwith the x, y, z coordinate system of FIG. 14 with its x-y origin 312 soplaced as to be entered on the platform. Of course, this is just aconvenient place to put it and it could be centered elsewhere as well. Apoint 314 represents the center of the head of the viewer in the helmet104 in the image space 102 (see also FIG. 7). It may be assumed forpurposes of the example that a distance Z₁ between the points 312, 314is constant. This is an approximation good for the case where only threeattitudinal axes are positively measured and/or controlled, and notposition, as in the exemplary embodiment of FIG. 7. It will thereforealso be assumed that there is no positional (i.e., translatory) movementof the head origin 314 in the x, y and z directions with respect to theorigin 312 and that all the distances x₁, y₁ and z, are thereforeconstant as well. It should be well understood, however, that more orless than three axes may be positively measured and controlled,including translatory position of the head. In the exemplary embodiment,where only head attitude is of interest and is measured in three axesand similarly controlled in those axes, it may be assumed for purposesof approximation, that the origin of the head of the cameraman/viewer ispositionally stationary, i.e., is not translating with respect to theorigin 312 of the platform 234. However, it should clearly be understoodthat this may not and need not be the case and that the distances x₁,y₁, and z₁ between the platform 234 and the origin 314 can be positivelycontrolled and used as a parameter in determining the image to beprovided to the eyes of the viewer. This of course implies a sensing ofa similar parameter in the object space as previously explained inconnection with the steps of FIG. 9, for example. This applies to thedistances x₁ and y₁ in the x and y axes separately or equally, asdesired.

In any event, it may be assumed for purposes of approximation for theillustrated embodiment that the head origin 314 is positionally immobileand that, as a result, x₁, y₁, z₁ are constant and that the onlyvariables are the sensed pitch, roll, and yaw parameters, as indicatedin FIG. 17 by angles P(y-z plane), R(x-z plane) and Y(x-y plane),respectively. It may be assumed for purposes of visualization andconvenience of illustration that the not illustrated viewer in a rest orzero position is sitting or standing erect with his head centered atpoint 314 and facing in the positive y direction.

If it is desired to move the viewer's head so as to be guided to assumea particular pitch and roll attitude, such as dictated by the signals onthe line 230 of FIG. 7, the plate 153 of FIG. 14 will be moved into aposition so as to position the universal joint in the gear 290 at aparticular position such as centered at a position 316 shown in FIG. 17.A shift of minus Δx and positive Δy, as shown, will effect such achange. This will cause the rod 296 to be aligned along an axis 318intersecting the points 314 and 316 and the viewer's head will assumethe desired pitch and roll attitude at particular pitch and roll anglesP, R. Trigonometry can be used to calculate the Δx and Δy movements thatwill result in the desired pitch and roll angles. For example, if the Pangle is determined in the step 152 of FIG. 9, as previously described,the step 154 calculates the Δy needed to obtain the desired P anglebased on the fixed distance z, and the desired angle P. I.e., Δy=z₁tanP.The actuator 232 c of FIG. 14 then causes the platform 153 to move inthe positive y direction by the calculated length Δy.

Corresponding to the method of FIG. 9 for the pitch axis, it will berealized that the method of FIG. 9 can be adapted to apply as well tothe roll axis. The illustrated roll angle R is obtained, e.g., by anegative Δx translation of (Δy)(tanR)/(sinP) or, equivalently,Δx=z₁tanR. The actuator 232 a of FIG. 14 may then be used to move theplatform a distance of Δx in the negative direction.

The third degree of freedom, i.e., the yaw axis may be controlleddirectly by means of the actuator 232 b of FIG. 14. By turning the gear288, the gear 290 and the universal joint within are rotated by theangle Y and the head of the passive viewer is accordingly rotated by theangle Y.

Upon induced movement by the apparatus of FIGS. 14 & 17, for theillustrated embodiment, the viewer's head need not stay preciselycentered or positioned at point 314 as this is just an approximation.The attitudinal changes of the viewer's head induced by the apparatus ofFIGS. 14 & 17 will naturally be accompanied by some small translatorymovements due to the interaction of the head, neck, and body of theviewer even though the body be relatively stationary. In other words,the rod 296 is able to slide within the U-joint and the viewer can havesome control over the translatory position of his bead along the axis ofthe rod 296. Nevertheless, other embodiments are possible in which thepassive viewer's bead position is more precisely controlled.

Similarly, it will of course be realized that the known principles oftransformation of coordinate systems may be employed to transform andtranslate the sensed signals in the object space of FIG. 7 into thecoordinate system of FIG. 17, as desired.

It should also be understood that the attitude control signals on theline 230 of FIG. 7 change from time to time to cause the viewer's headto assume various different attitudes. At the same time, it should beunderstood, correspondingly different attitudinal views of the depictedscene are presented on the display 106 so as to provide a harmoniouslychanging viewpoint for the passive viewer. In other words, the viewer ismade to think that he is viewing a real world object space fromdifferent angles as his head is guided in changing attitudes. Thesedifferent views may be provided by the cameras 108, 110 on thecameraman's head but can be generated by other means 86 such asanimation or computer generated imagery.

As mentioned above, in connection with FIGS. 7 & 13, the signalprocessor or encoder 116 receives at least one video signal from atleast one camera and, in the embodiment illustrated, receives two videosignals 112, 114 from left and right video cameras 108, 110 to provide astereoscopic video signal to the image space. These signals that areencoded by the signal processor or encoder 116 may be provided to thedecoder 220 in the image space 102 so as to provide a viewer withstereoscopic images of the object space. These may be provided byseparate displays, one for each eye, or may be provided by the samedisplay alternately, using light shutters, as known in the art of stereotelevision.

As described above, the viewer can be drawn even more deeply into theexperience of the cameraman (or a computer generated version thereof) byhaving his visual axes induced or guided to emulate those of thecameraman. This is not necessary but represents an enhancement of theinvention. This is done by monitoring the visual axes of one or moreeyes of the cameraman in order to determine the direction of his gaze.That information is then used to produce each image in such a way thatit has nonuniform informational content over its expanse. In otherwords, a portion of each image will have more or less informationcontent concentrated or highlighted therein, in order to draw theattention of the viewer thereto. The particular portion of eachsuccessive image that has this extra level of informational content orhighlighting will be changed between successive images such that itmoves about within the field of view of the viewer according to thedirection of the cameraman's gaze within his field of view and thepassive viewer's eyes will naturally move about in emulation thereof.This sort of a passive viewing of an image nonuniformity, e.g., of ahigh concentration portion of the image is disclosed in detail incopending U.S. patent application Ser. No. 08/001,736, now U.S. Pat. No.5,422,653.

As also mentioned, FIG. 13 shows a pair of eye monitors 169 a, 169 bthat provide sensed signals on the lines 240 a, 240 b to left and rightcamera controls 242, 244 which in turn provide left and right cameracontrol signals on the lines 246, 248 for controlling the imagenonuniformity or high informational content portion of the respectiveimage signals on the lines 112, 114.

The nature of the nonuniform images can be selected according to thedemands of a given application and need not be of any particular type.They may be of the type shown, for example, in U.S. Pat. No. 3,953,111by a nonlinear lens or as shown in U.S. Pat. No. 4,028,725 or 4,405,943or 4,513,317 (see FIGS. 2, 3 & 4 in particular) or 3,507,988 or as shownin the above mentioned copending application U.S. Ser. No. 08/001,736(nonuniform resolution), now U.S. Pat. No. 5,422,653 or as describedbelow by images having nonuniform dynamic range for the individualpixels therein. Nonuniform resolution images may be provided in analogor digital fashion as described in the copending U.S. patent applicationSer. No. 08/001,736 (now U.S. Pat. No. 5,422,653) in connection withFIGS. 7( a), 7(b), 8-12, 13(a)-(c), and 14 and as described in theaccompanying text thereof beginning at page 29, line 3 through page 51,line 14 which is hereby expressly incorporated by reference. Similarly,a nonuniform dynamic range technique may be used in lieu of nonuniformresolution, particularly for digital embodiments. In such a case, asmall group of pixels within the total picture are selected to be sensedand/or encoded with a greater number of levels or shades of gray thanthe larger remainder portion of the total picture. For these pixels,more digital bits are used so as to achieve the greater number oflevels. The position of the small group within the total image may bemoved as between successive images or within a single image according tothe cameraman's monitored eye movements within the cameraman's field ofview. To carry out the nonuniform imagery, regardless of type, all thatis required is that the portion of each successive image that containsthe nonuniformity content be presented in such a way as to draw theattention of the viewer's gaze so that the nonuniform portion falls onthe viewer's fovea.

For example, FIG. 18 shows an object space 320 comprising the interiorof St. Ignatius in Rome. A cameraman such as the cameraman of FIGS. 7and 13 is located in the object space 320 with a helmet for gatheringmonocular or stereoscopic images. Assuming the cameraman is seated inthe church, as the cameraman moves his head to look about by makingattitudinal (pitch, roll, and yaw changes, the camera or cameras pick upthe images 322, 324, 326, 328 shown over a short period, e.g., a secondor two more or less. It may be assumed that for the first image 322, thecameraman is seated, with his head level and pointing straight ahead,e.g., pointing in the +y direction of FIG. 17. Due to the attitudemonitors of FIG. 7 and the control signals developed therefrom asexplained in connection with FIG. 9, the display apparatus in the imagespace will also be oriented so as to cause the passive viewer's head tobe similarly oriented so as to see the same image 322 with his head inthe same attitude as that of the cameraman.

The images may be provided with uniform resolution. In the particularinstance illustrated, however, for the acquired image 322, thecameraman's visual axis is not directed straight ahead but off to theleft as monitored, e.g., by the oculometer 166 of FIG. 7 or 180 a, 180 bof FIG. 12 or monitors 169 a, 169 b FIG. 13. As a result, the displayedimage has a small area 330 that has higher image informational contentthan the rest of the image which is shown with fainter lines to indicatea lesser degree of image informational content. The area 330 may be ofhigher resolution, dynamic range, or the like. The passive viewer'svisual axis is naturally drawn to be directed on the area 330 forviewing by the fovea of the passive viewer's eye. As a result, theviewer emulates with foveal viewing not only the head movements of thecameraman but also his eye movements.

It should be mentioned that if a variable magnification device such asthe device 239 of FIG. 13 is used, the magnification of the image 132can be changed according to the control signal on the line 238 to changethe accommodation of the eyes of the passive viewer. An example of suchis shown in FIG. 20, where a pair of eyes are shown fixating first at astraight ahead point at a far distance and then at a closer point butoff to the side. Thus, the images 134, 136, 138 to be described belowcan be viewed with differing accommodation, especially to the extentthat they represent objects at differing distances as indicated by theeye monitor or monitors 169 a, 169 b. In this way, the variablemagnification device is responsive to light from the light source and toa variable magnification control signal, for changing the apparentdistances of the images. It should be realized that the changing of theapparent distances can also be accomplished by changing the actualdistance of the display, in response to a control signal. It should alsobe realized that the degree of accommodation experienced in the eyes ofthe viewer can be coordinated with convergence of the eyes so as tomaintain a selected relationship therebetween, preferably a normalrelationship, e.g., as shown in FIG. 19 of copending application Ser.No. 08/462,503.

The cameraman next moves his head up and to the left, i.e., executes aclockwise movement (pitch) in the y-z plane and a counterclockwise(viewed from above) movement (yaw) in the y-x plane of FIG. 17. Theacquired image 324 is the result. It will be observed that thecameraman's visual axis has changed its point of attention within theobject space to a small area 332 on a lower right hand side of “his’field of view. The display apparatus in the image space of FIG. 7 willcause the passive viewer to execute similar attitudinal head movements.Similarly, the passive viewer's eyes are naturally drawn to the smallarea 332 for viewing by the fovea of the passive viewer's eye. If thesmall area 142 is or should be represented at a different distance thanthe area 140 then the variable magnification device may be used to causea differing accommodative response.

The cameraman next moves his head up and to the right, i.e., executes aclockwise movement (pitch) in the y-z plane and a clockwise movement(yaw) in the y-x plane and acquires the image 326. the cameraman'svisual axis in this case is still directed to the right but slightlyabove to a small area 334 of higher image informational content. Thedisplay apparatus in the image space of FIG. 7 will cause the passiveviewer to execute similar attitudinal head movements. Similarly, thepassive viewer's eyes are naturally drawn to a small area 334 forviewing by the fovea of the passive viewer's eye. And if a change inaccommodation is appropriate, such can be controlled according to thedevice 239 of FIG. 13.

Finally, the cameraman next moves his head further up and to the right,i.e., executes a clockwise movement (pitch) in the y-z plane and aclockwise movement (yaw) in the y-x plane and acquires the image 328.The cameraman's visual axis in this case is still directed to the rightbut slightly down to a small area 336 of higher image informationalcontent. The display apparatus in the image space of FIG. 7 will causethe passive viewer to execute similar attitudinal head movements.Similarly, the passive viewer's eyes are naturally drawn to the smallarea 336 for viewing by the fovea of the passive viewer's eye andappropriate accommodative changes can be induced. The illustrated headmovements then continue in a similar way.

Although no roll movements (in the x-z plane) have been illustrated (asa tilt) in FIG. 18, such are of course contemplated, according to theinvention. It should be realized that the images gathered in the objectspace and presented in the image space succeed each other at a veryrapid rate, e.g., 30, 60, 120 or even more frames per second. Thus, forthe illustration of FIG. 18 there will likely be a multitude of imagespresented beyond the few shown so that the sequence or presentation ofimages is much smoother than indicated.

As already indicated in connection with FIG. 7, the images gathered inthe object space may be stereoscopic. Such may be presented by thedisplay in the image space of FIG. 7 in any convenient stereoscopicformat of uniform or nonuniform resolution presentation. For a narrowfield of view such as shown in FIG. 18 (e.g., on the order of 30-45degrees) the stereopair images are completely overlapped and still onlycover half of the full 90 degrees of binocular vision of the humanvisual process. An Asher-Law stereoscope as taught in FIGS. 20-22 ofcopending application Ser. No. 08/462,503 would be a suitable means ofpresentation as described at page 64, line 24 through page 68, line 8which is hereby incorporated by reference.

As shown in copending application Ser. No. 08/462,503, based on U.S.Ser. No. 08/1,736, now U.S. Pat. No. 5,422,653, a stereopair may bepartially overlapped as described at page 57, line 6 through page 63,line 20 and as shown in FIGS. 17 and 18( a)-(d) with separate verynarrow field of view areas of high image informational content in theseparate left and right views coinciding and moving about togetherwithin an area of binocular overlap. Such an approach is particularlyappropriate where an overall very wide field of view is presented, e.g.,wider than 90 degrees, i.e., where the areas outside 90 degrees aremonocular, in imitation of the human field of view. Such wide angleimages can be acquired with a high degree of detail over the full fieldof view of the human visual apparatus using an array of image sensormodules such as shown in U.S. Pat. No. 4,323,925. Or, a pair of virtualretinal displays such as shown in U.S. Pat. No. 5,467,104, 5,596,339, or5,574,473 can be used. The degree of detail achieved, however, ischiefly important in the area of stereoscopic foveal viewing.

For example, as shown in FIG. 19, a sequence of four stereoscopic images338, 340, 342, 344 similar to the sequence of FIG. 18 are shown, eachcomprising corresponding overlapping left and right halves 338 a, 338 b;340 a, 340 b; 342 a, 342 b; 344 a, 344 b. Each half represents the fieldof view of a respective left or right eye. Each can have a horizontalfield of view of as wide as 140 degrees, approximately corresponding tothe full human monocular field. An inverted triangular shaped area ofoverlap of each pair is viewed stereoscopically, i.e., binocularly andcan be as wide as 90 degrees horizontally corresponding to the fullhorizontal extent of human binocular overlap. As in FIG. 18, the imagessequence from the bottom to top of the Figure with an accompanyingsequence of head movements. Within each triangular shaped area ofbinocular overlap in each pair are two overlapping areas of highconcentration image information 338 c, 340 c, 342 c, 344 c. Since theseare overlapping in the Figure, they appear as one. They are similar tothose shown in FIG. 18 except for being indicative of binocular fovealfusion.

For thus viewing stereo images by two eyes, the light control signalcontains information for providing light for viewing stereo imageshaving a portion with high informational content and a portion with lowinformational content and wherein the portion with high informationalcontent changes position within the images for foveal viewing by the twoeyes by following the position changes. If it is desired to include oneor a pair of variable accommodation devices, such can be used to changethe accommodation of the two eyes either together or independently.Moreover, the light control signal can contain information for providingthe light for viewing stereo images having a portion with highinformational content and a portion with low informational content andwherein the portion with high informational content changes positionwithin the images for foveal viewing by the two eyes by following theposition changes at correspondingly changing convergence, i.e., atcorrespondingly changing visual fixation points. In other words, thevisual axes of the eyes will intersect at varying points in athree-dimensional image space. The accommodative changes to be inducedat each such fixation point can be controlled for each eyeindependently, or for both eyes to the same degree. Independenttreatment is more important at close fixation point distances. This isbecause the distance from one eye to the point of fixation may bemarkedly different from that of the other. Such occurs primarily when aclose fixation point is markedly off to one side or the other of theviewer's field of view.

It may be the case that eye tracking in the object space is not desiredbut that an emphasis of the binocular area of overlap is nonethelesswanted. In that case the area of high resolution emphasis can beenlarged, e.g., as shown in FIG. 21 to cover the entire area ofbinocular overlap in a sequence similar to those shown in FIGS. 18 & 19.In such a case, no matter where the viewer chooses to fixate within thebinocular overlap area he will view the imagery with a high level ofresolution. This would eliminate the need for controlling the positionof a small area of high informational content by eye monitoring or thelike while at the same time reducing the informational content in theareas of nonoverlap of the monocular fields, albeit with the loss offull control of simulation of active percepts for passive viewing.Similarly, a central, fixed area of each of the images FIG. 18 could berendered with higher informational content or highlighted in a selectedmanner.

As already suggested above, it may also be useful to provide the imageto the viewer at various apparent distances as shown for example incopending application U.S. Ser. No. 08/25,975 at page 6, line 25 throughpage 32, line 10 by means of a variable magnification device such asshown (but not limited to) in copending application having U.S. Ser. No.08/1,736 at page 24, line 23 through page 29, line 2 in connection withFIGS. 3-6 thereof, and at page 70, line 11 through page 72, line 2 inconnection with FIGS. 26-35 thereof, all of which is hereby expresslyincorporated by reference.

For a virtual retinal display, on the other hand, all the objects can berepresented at the same time at their various apparent distances byappropriate defections of the scanning light beam, e.g., by actuatinglenses and mirrors for providing the light beam from a selectedtrajectory for each pixel impinging on the retina of the viewer. In thatcase, according to the present invention, the eyes of the passive viewercan be induced to follow a selected series of visual fixations by“painting” only a small portion of the scene with a fine light beam andthe rest a coarse beam. The viewer's eyes will then be induced to turntheir foveas in the direction of the fine portion of the scene.

In connection with any of these various types of images, as alsosuggested above, it may be desired to present the images to the eyes ofthe viewer at various apparent distances in such a way as to preserve aselected relationship such as, but not necessarily, a normalrelationship between accommodation and convergence in the eyes of theviewer. Such is shown, for example, beginning with FIG. 19 and asdescribed beginning at page 63, line 21 through page 68, line 8 and infurther connection with FIGS. 20-22 of the above mentioned U.S. Ser. No.08/1,736, now U.S. Pat. No. 5,422,653, which is incorporated byreference. For a virtual retinal display as modified as described above,the effect is automatic, assuming the representative of apparentdistances is accurate. Merely by being induced to fixate on a point atthe desired distance, the eyes of the passive viewer accommodate andconverge harmoniously, in a normal manner.

The helmet mounted display of the image space of FIG. 1 can take theform as shown, for example, in FIGS. 23-25 of U.S. patent applicationSer. No. 08/1,736 as described beginning at page 68, line 9 through page70, line 10, now U.S. Pat. No. 5,422,653, which is incorporated byreference.

Similarly, although not disclosed herein, it should be realized that anaudio component of the video signal on the line 226 may be provided aswell by placing microphones in the object space such as positioned instrategic locations on the cameraman's head. Similarly, speakers may beprovided in the image space and located in analogous positions about thehead of the viewer for receiving decoded audio signals from the decoder.I.e., the speakers may be placed strategically about the helmet forreproducing sound as if from three dimensions as heard by the cameraman.Such is shown in detail in (copending application Ser. No. 08/1,736)U.S. Pat. No. 5,422,653 at col. 29 (page 52), line 4 (18 through col.(page 57), line (5) in connection with FIG. 16 and at col. (page 68),line (14) through col. (page 69), line (7) in connection with FIG. 23,all of which is hereby expressly incorporated by reference.

Although most of the embodiments shown thus far show the user in astanding or otherwise semi-erect position in a chair, it should berealized that the invention is applicable to a more relaxed position forthe user. For instance, FIG. 22 shows a support 402 for supporting thebody of a user and more particularly for supporting a moveable headrest404 for supporting the back of the head 406 of the user in executinghead movements while viewing images provided by a display 408 from achanging direction, i.e., a changing “direction-of-view,” according tothe invention. In other words, the direction, i.e., attitude of the headof the viewer actively changes or is passively changed for viewingimages from a correspondingly changing direction. In the case where themoveable headrest is moveable by the user actively changing thedirection of his head, i.e., under his own volition, the movements aremonitored by a sensor 410 for the purpose of providing an input signalon a line 412 to a reality engine (see FIGS. 32 and 33) for selectingthe images according to the changing direction. In the case where theattitude of the head of the user is passively changed, the moveableheadrest is moveable by an actuator 414 to change the“direction-of-view” of the head of the user in response to a commandsignal on a line 416 from a reality engine (see FIGS. 34 and 35) withthe images changing their direction-of-view correspondingly, accordingto an image signal from the reality engine. Although the display 408 isshown adjacent the head 406, it should be realized that it can be apartfrom the head.

FIG. 23 shows a user 417 reclining on a support in the form of arecliner 402 a equipped with a moveable headrest 404 a with a rod 20supported by a support 418 that is attached to or part of the support402 a. The support 418 may comprise brackets 422 through which the rod420 is threaded. The rod may come to rest or be fixedly terminated on astop 424. The head 406 a of the user in a reclining position rests onthe headrest 404 a for viewing images from a changing direction-of-viewprovided by a display 408 a which may be any kind of display. One of themany types of display that may be used is a head mounted display such asshown in U.S. Pat. No. 5,671,037. Although the recliner 402 a is shownas a stationary support, it can be of the type preferably as shown inU.S. Pat. No. 5,695,406 but also as shown in other chair type simulatorssuch as, among others, U.S. Pat. Nos. 6,056,362, 5,678,889, 3,628,829,5,490,784, as well as others filed after the Mar. 13, 1999 US parent(60/124,642) of the present application such as U.S. Pat. No. 6,152,828or 6,113,500. In such cases, the translations determined in the steps ofFIG. 9, particularly accelerations associated therewith, can be used tocontrol the chair or other platform.

It should be realized that although the moveable headrest is shownsupported by a support 402 which may also support the body of the user,the body of the user need not be supported by the support 402 but may besupported in some other way. In other words, the moveable headrest mayequivalently be supported by a support that is different from thesupport provided for supporting the body of the user. For instance, theheadrest might be wall-mounted and the user support positioned nearby toallow the head of the supported user to rest on the headrest in the sameway as shown above while the user may lean against or stand next to thewall.

For another instance, FIG. 24 shows a user 417 b secured in a standingposition on a support 402 b in the form of a human activity simulatorsuch as shown in U.S. Pat. No. 5,792,031. The simulator is equipped,according to the present invention, with a moveable headrest 404 b forsupporting the head of the user while viewing images provided by adisplay 408 b from a changing direction.

In yet another instance, FIG. 25 shows a user 417 c seated on a supportin the form of a chair 402 c equipped with a moveable headrest 404 cwhile viewing images provided by a display 408 c from a changingdirection, according to the invention. Although the chair 402 c is shownas a stationary support, it can be of the type shown in U.S. Pat. No.5,642,302, modified appropriately to be continuously positionable, as inthe recliner of U.S. Pat. No. 5,695,406.

FIG. 26 shows a moveable headrest 404 d with a pivotal support assemblyfor supporting the head 406 d of a user in executing head movements witha changing direction-of-view while viewing images from a correspondinglychanging direction-of-view. Except for the display 408 d, an actuator426, a sensor 428 and a supporting bracket 430, the hardware for theheadrest structure shown in FIG. 26 and described below is taken fromU.S. Pat. No. 5,791,735 of Helman entitled “Headrest assembly with useractuated pivotal support assembly.” The Helman headrest is forattachment to a wheelchair seating system for use in supporting andassisting with movements of the head of a patient with weak neckmuscles. It includes a cushioned backpad 432, a pair of laterally spacedapart cushioned side pads 434, and a mounting assembly 436 locatedsubstantially to the rear of said headrest. The mounting assembly 436enables the backpad and the sidepads to rotate together as a unit abouta vertical axis located forward of the mounting assembly. The mountingassembly is formed such that the vertical axis of rotation of theassembly substantially coincides with the spinal column of the humanpatient seated in the wheelchair. The rotational range of backpad 432and sidepads 434 is manually adjustable and limited to a prescribedrange. A force is formed with a rubber band 435 resisting rotationaldisplacement in said backpad 432 and sidepads 434 returning saidheadrest to a null, forward facing orientation. The strength of theforce increases with a corresponding increase in rotational displacementof the headrest. For the purposes of the present invention, the headrestassembly can be used to support the head of a user while viewing imagesfrom a controlled location.

To enable cushions 432, 434 to be positioned so that they best fit theparticular user's head 406 d, pivoting assemblies 438 connect thesupport cushions 432, 434 to structural members 440, 442, respectively.The pivoting assemblies 438 are composed off a ball and socket typejoint which allows pivoting of cushions 432, 434. This pivoting actionallows cushions 432, 434 to be tilted and oriented such that they bestfirmly contact the head of the particular user supported as shown e.g.in FIGS. 23-25, thus accommodating different shapes and sizes of headsof different users. It should be realized that the left sidepad 434 isshown unpivoted in FIG. 26 in order to show the pivoting assembly. Witha human head resting against the sidepad 434 it will be tilted downwardto engage the bottom of the skull of the head 406 d as suggested by thedashed line 437. An additional pivoting assembly 444 connects extensionmember 440 to the mounting assembly 436. The mounting assembly 436extends rearwardly of the backpad 432 and the sidepads 434 to a supportmounting structure 446. Pivoting assembly 444 is composed of a ball anda socket type joint. Pivoting assembly 444 permits cushion 432 to bemoved in a back and forth direction thus enabling even greaterflexibility in positioning cushion 432 to the particular shape of theuser's head. Side cushions 434 may also be adjusted in a back and forthdirection since structural members 442 are received in a vice typebracket 448 which holds structural members 442 in place. Bolts 450 inbrackets 448 can be loosened such that structural members 418 can beslid the slightly forward and backward, thus giving another dimension ofmovement in positioning cushions 434. After adjusting structural members442 to the particular head shape of the user, bolts 450 are tightened inplace fixing the side cushions 434 in position. Operating together,cushions 432 and 434 provide comfortable support to the rear and sideson the user's head. Balancing for the user's head is also provided bythese supporting cushions.

The Helman headrest assembly provides support and balancing to theuser's head not only when it is stationary, but also through limiteddegrees of motion. To accomplish this, the supporting cushions 432, 434must be able to move with the rotation of the head. The mechanism toaccomplish this function is found in the design of mounting assembly 436which guides the motion of the cushions 432, 434. Mounting assembly 436comprises a plurality of four forwardly extending arms 452, 454, 456,458 which together comprise a linkage assembly. The movement of theseforwardly extending arms with respect to one another allows the user tohave freedom to rotate his head to a limited degree. The method offunctioning of the forwardly extending arms 452, 454, 456 and 458 isexplained in more detail in the above-mentioned U.S. Pat. No. 5,791,735of Helman which is hereby incorporated by reference. Suffice it to saythat connectors 460, 462, 464 separate the various forwardly extendingarms 452, 454, 456, 458 and enable these arms to pivotally rotate freelyabout the points through which the respective connector passes.

As mentioned above, the user is equipped with a display 408 d, accordingto the present invention, which provides images from a changingdirection-of-view in correspondence with a changing direction-of-view ofthe head 6 d of the user. The user's head is supported and/or guidedfrom the rear and the user can consequently assume a relaxed posture.The display need not be of the goggle type shown in FIG. 26, but may beof any type. FIG. 27 shows the user with his head 406 d oriented in astraight-ahead looking direction for viewing a central rotund part ofthe U.S. Capital Building in silhouette with a certain field-of-view. InFIG. 28, the field-of-view has been shifted left for viewing theleft-hand side of the U.S. Capital Building. In FIG. 29, thefield-of-view has been shifted right for viewing the right-hand side ofthe U.S. Capitol Building.

Depending on the design, the display can be used actively only,passively only, or in a dual mode version either actively or passively.FIG. 26 shows a design in which the headrest assembly of Helman can beused either actively or passively. This is accomplished by providingboth a motor 426 and a sensor 428 on the bracket 430 which is rigidlyattached to the mounting structure 446. The shaft of the motor andsensor may be axially coupled and fixed to the arm 454 for rotating thearm 454 about the common axis of the motor and sensor. This causesrotations of the other arms 452, 456, 458, which are shown in moredetail in U.S. Pat. No. 5,791,735 of Helman. The motor 426 may be astepping motor, a servo motor, or the like, for use in a passive mode ofoperation to actuate the headrest assembly in executing headrestmovements such as illustrated in FIGS. 27-29 for guiding the head 406 dof the user. In that case, the sensed output signal from the sensor 426may be unutilized (open loop control) or may be used as a feedbacksignal (closed loop control). An open loop control is shown in FIG. 30with the command signal on the line 416 provided to a simpleproportional amplifier that in turn provides an amplified output signalon a line 416 a to the actuator 414. On other hand, the sensor 428 maybe used in an active mode of operation to sense movements of theheadrest assembly such as illustrated in FIGS. 27-29 as actuated by thevolitional movements of the user's head 6 d. The sensor may be an RVDTor rotary potentiometer, for instance, for sensing angular displacement.A closed loop control is shown in FIG. 31 with the command signal on theline 416 provided to a summing junction where the sensed signal on theline 412 is subtracted therefrom. A difference signal is provided by thesummer to a compensator such as a proportional-integral (P-I)compensator that in turn provides a compensated output signal on a line416 b to the actuator 414 such as the motor 426.

FIG. 32 shows hardware setup for active use of the moveable headrest,i.e., with the user moving his head at will. A sensor 410 a provides asenses signal on a line 412 a to a local reality engine 470 a which, inresponse thereto, retrieves an image sequence from a memory thereinhaving a plurality of such stored sequences. The signal on the line 412a is comparable to the signal on the line 28 of FIG. 3 of U.S. Pat. No.5,644,324. The retrieved sequence is provided on a line 472 a to adisplay 408 a for viewing by the active user. The reality engine 408 amay be embodied in a local computer or a remote computer accessiblethrough a network 474 b, as shown in FIG. 33. In the example of FIG. 32,the active user moves his head and the headrest follows. The sensor 410a senses the angular rotation of the headrest and provides the sensedsignal on the line 412 a to the reality engine 470 a which in turnprovides the retrieved sequence on the line 472 a to the display 408 a.The retrieved sequence of images are taken from different directions ofview, corresponding to the viewer's active head movements. The imagesare presented from differing directions of view according to the activeuser's head movements to make him feel that he is moving his head andviewing the virtual world in the same way he views the real world. FIG.33 is the same as FIG. 32 except that the reality engine 470 b is remoteand is accessed via a network 474 b. The reference numerals are similarto those of FIG. 32 except with the “be” suffix.

FIG. 34 shows passive use of the moveable headrest with an actuator 414c moving the headrest and hence the user's bead in response to a signalon a line 416 c from a local reality engine 470 c. The signal on theline 416 c is comparable to the signal on the line 20 of FIG. 1 of U.S.Pat. No. 5,734,421. The sensor 490 c provides the sensed signal on theline 412 c to the local reality engine 470 c as a feedback signal, forclosed loop control. For a headrest that is only to be used for passiveusers, it should be realized that a sensor is not absolutely necessarysince open loop control of the headrest will work. In the case ofpassive use, the reality engine retrieves a single, preplanned imagesequence from a memory therein, in response to a start command signal ona line 476 c. The start command signal on the line 476 c can originatewith the passive viewer pressing a button, voicing a speech command,having his eyes monitored, by some combination of such commands, or thelike. The retrieved sequence is provided on a line 472 c to a display408 c for viewing by the passive user. The reality engine 470 c may beembodied in a local computer or a remote computer accessible through thenetwork 474 d, as shown in FIG. 35. FIG. 35 is the same as FIG. 34except the reality engine is remote and accessed via a network 474 d.The reference numerals in FIG. 35 are similar to those of FIG. 34 excepthaving the suffix “d”.

Referring back to the variable magnification device 239 of FIG. 13, itshould be pointed out that the provision of successive images to the eyeat varying apparent distances for viewing with correspondingly varyingfocus (accommodation) creates a granularity problem. With increasingfocus, because of the limitations of man-made imaging technology, thereis not any increased level of granularity available for inspection.Therefore, there can be a problem with the simulated reality of theimagery under increased focus. As suggested, the granularity of a givenstatic man-made image of a real object is only as good as that of theimaging technology used to acquire and present it. Closer inspectionwith a magnifying glass or other aid to eyesight does not ultimatelyreveal any deeper granularity but only the limitations of the imagingtechnology used. This is not usually a problem for images in books,movies and other conventional media.

On the other hand, the granularity of real objects is unlimited as faras the human eye is concerned. Considering the eye itself, withincreased focus, more detailed granularity of objects is alwaysrevealed. Moreover, with technological aids to the eye, e.g., themagnifying glass, the optical microscope, the electron microscope, andother tools, smaller details are always revealed.

Referring back to the cameras 108, 110 of FIGS. 7, 12 and 13, these canbe equipped with lenses that vary the focal length according to acontrol signal. For instance, as shown in FIG. 36, one of the cameras108, 110 of FIGS. 7, 12 and 13 is shown collecting images of a scene 432illuminated by a light source 434. An eye 436 of the cameraman of FIG. 7is shown using an eyepiece 438 for viewing the scene 432 beingphotographed. A sensor 440 which may take the form of one of the eyemonitors 169 of FIG. 13 such as does an eye accommodation sensor sensesthe accommodation of the eye 436. The sensor 440 provides a sensedsignal on a line 442 to an optic control 444. The optic control 444provides a camera optic control signal on a signal line 446 to, e.g., amotorized camera optic 448. A motorized optic is for example only andcould take other forms. The optic control 444 causes the motorized optic448 to focus on the scene 432 at differing focal lengths according tochanges in the accommodation, direction, or the like, of the eye 436 asdetected by the sensor 440. The optic 448 casts rays 450 on to an imagesensor 452 that provides a video image signal on a line 453 to acombiner 454. It combines, e.g., in a time division multiplexed way, theimage signal on the line 453 with the signal on the line 442 to form thevideo data signal on the line 118 of FIG. 7. As explained above, thesignal on the line 442 could be provided in parallel on a separatesignal line alongside the signal on the line 222. In that case, it wouldonly carry video information.

Referring now to FIG. 37, an apparatus 510 is there shown for showingimages to an eye 512 of a passive viewer such as the eye 14 of FIG. 1 oran eye of a user of the devices of FIG. 2A, 2B, 3, 4, 7, 23, 24, or 25.A video signal is received on the line 118 by a control 516 similar tothe controls 224 and 205 of FIG. 7, for example. The video signalcontains image information which is decoded and provided on a signalline 518 to for instance an image projector 520 which projects imageswith first light rays 522 to a first optic 524. The first optic 524 mayfor instance be a lens that is under the control of a control signal ona line 526 from the control 516. The control signal on the line 526 isdecoded by the control 516 from the video signal on the line 118. Thefirst optic 524 refracts or otherwise bends the light rays 522 intosecond light rays 527 that are projected on to a screen such as atranslucent screen 28 to form images of different sizes, i.e., that fillthe screen 28 to a greater or lesser extent as shown in FIGS. 38-40, asdiscussed below. The signal on the line 118 can be provided in manydifferent ways. For instance, it should be realized that the signal onthe line 118 need not be a single line (which implies some form ofmultiplexing) but could be two or more signal lines.

It will also be realized from the foregoing that the control signal onthe line 526 changes the projected first light rays 522 by means of theoptic 524 according to changes detected in the cameraman's eye 436 ofFIG. 36 by the sensor 440. In other words, the signal on the line 442from the sensor 440 is not only used to control the optic 448 in theobject space of FIG. 36, but is also used to control the optic 524 inthe image space of FIG. 37 after transmission to the image space overthe signal line 118. The nature of the change in the projected firstlight rays 522 is manifested by the manner in which the light rays 527are projected on to the screen 528. It should be realized that the rays527 would be reflected from a mirror before being sent to the screen 528which could be reflective rather than transmissive. Such would result ina folded embodiment rather than the straight system shown. The examplesof FIGS. 38-40 have already been referred to. If the eye 436 of FIG. 36is detected by the sensor 440 viewing the scene 432 with a long focaldistance, such as infinity, the optic control 444 causes the optic 448to focus on a correspondingly long distance. The optic 448 focuses thescene 432 at infinity and projects the details of the scene with a widefield of view on to the image sensor 452. Consequently, the availablesensor 452 pixels are spread over a relatively wide field of view. Inother words, the granularity of the image is spread over a wide field ofview. FIG. 38 shows the second light rays 527 projected to form an image554 that fills or almost fills the entire area or extent of the screen528.

If the eye 436 of FIG. 36 is detected by the sensor 440 viewing thescene 432 with a lesser focal distance such as an intermediate focaldistance, the optic control 444 causes the optic 448 to focus at acorrespondingly intermediate distance. The optic 448 focuses the scene432 at the intermediate distance and projects the details of the scenewith an intermediate field of view on to the image sensor 452.Consequently, the granularity, i.e., the available sensor pixels arespread over a relatively intermediate field of view. FIG. 39 shows thesecond light rays 527 projected to form an image 556 that only partiallyfills the entire extent of the screen 528.

If the eye 436 of FIG. 36 is detected by the sensor 440 viewing thescene 432 with a short focal distance, the optic control 444 causes theoptic 448 to focus at a correspondingly short distance. The optic 448focuses the scene 432 at a correspondingly short distance and projectsthe details of the scene with a narrow field of view on to the imagesensor 452. Consequently, the available sensor pixels are spread over arelatively narrow field of view. Particular objects within the narrowedfield of view of FIG. 40 can be viewed with more granularity than thosesame objects could be with the granularity provided by that of FIG. 39and even more so than that of FIG. 38. FIG. 40 shows the result of thesecond light rays 527 projected to form an image 558 that only fills asmall extent of the entire extent of the screen 528.

Referring back to FIG. 37, as explained above, the second light rays 527are projected on to the screen 528 with different areas or extents 554,556, 558, all of which have the same total number of pixels. These aretransmitted as a third bundle of rays 529 to an optic 560. The advantageof this feature of the invention is that with the aid of the optic 560,the field of view of the eye 512 of the viewer can be fully occupiedwith all of these pixels even though the accommodation of the eyechanges. The total number of pixels can be spread over the full extentof the retina in all cases by a combination of changes in the focallength of the optic 560 and the accommodation of the eye 512. When theoptic 448 focuses in on a detail of the wider scene 432, it increasesthe granularity of the imaged scene in that area. At the same time, theoptic 524 causes the size of the image to be reduced on the screen 528as shown, e.g., in FIG. 39 or 40. In other words, when the focal lengthof the optic 448 is shortened to capture a narrowed field of view of thescene 432 with increased magnification, the granularity of that smallerportion of the imaged scene increases as manifested in a smaller area onthe screen 528 as controlled by the optic 524 and signal 526. At thesame time, the focal length of the optic 560 is controlled by a controlsignal on a line 564 from the control 516 to allow the eye 512 toaccommodate, i.e., to focus closer on to the scene with increasedgranularity in the area of interest. In other words, at the same timethat the control signal on the line 526 causes the optic 524 to reducethe extent to which the screen 28 is filled by imagery (see FIG. 39 or40), the signal on the line 564 causes the optic 560 to reduce the fieldof view provided for the eye 512, e.g., by increasing its magnification.Thus, the optic 560 refracts third rays 529 to provide fourth light rays564 in such a way that the eye 512 must change its accommodation so asto bring the image into focus on the reduced size imagery. This causesthe field of view of the eye 512 to be reduced but fully occupied withan up-close image while taking full advantage of the availablegranularity.

If the cameraman's eye 436 changes to a long view of the scene 432, asexplained above, the image 554 fills the screen 528 because of thecontrol signal on the line 526 causing the optic 524 to expand theextent to which the screen 28 is filled by imagery. At the same time,the signal on the line 564 causes the optic 560 to expand the field ofview provided for the eye 512, e.g., by reducing its magnification orincreasing its focal length. The eye 512 changes its accommodationaccordingly. In other words, when the control signal on the line 526causes the optic 524 to increase the extent to which the screen 24 isfilled by imagery, as in FIG. 38, the signal on the line 564 causes theoptic 560 to increase the field of view provided for the eye 512 evenfurther, e.g., by decreasing its magnification even more.

Similarly, although the invention has been shown and described withrespect to a best mode embodiment thereof, it should be understood bythose skilled in the art that various changes, omissions and deletionsin the form and detail of the foregoing may be made therein withoutdeparting from the spirit and scope of the invention.

1. A storage medium having successive images stored thereon forretrieval and presentation by a playback device to an eye of a viewer,said successive images including successive nonuniform images, eachsuccessive nonuniform image having a highly detailed component and alesser detailed component, the successive nonuniform images forsimulating visual perception of an observer of an object space wherein aposition of the highly detailed component is changed in selectedsuccessive nonuniform images such that a visual axis of the eye of theviewer with respect to a field of view of the selected successivenonuniform images is drawn to passively follow the position of thehighly detailed component so that the viewer perceives the successivenonuniform images as if looking through the eyes of the observer of theobject space wherein said successive images are computer generatedimages produced by a computer workstation with software or wherein saidsuccessive images are produced by a workstation on which software isused to carry out post-production work on images of real objects takenfrom a camera.
 2. (canceled)
 3. The storage medium of claim 1, whereinthe successive images are for presentation to said eye of the viewer atvarious apparent distances for viewing by the eye of the viewer withcorresponding accommodation so as to focus on the successive images atthe various apparent distances.
 4. The storage medium of claim 1,wherein said successive images are of objects in said object space forpresentation of different objects at differing focal lengths for viewingwith correspondingly differing accommodation, where light is projectedwith details of said scene over a widest field of view whenaccommodating at infinity and over correspondingly narrower fields ofview when accommodating at various distances closer than infinity, forproviding said successive images formed for display with a largest sizecorresponding to said widest field of view and with decreasing sizescorresponding to increasingly narrower fields of view such thatparticular objects in said scene are viewable with correspondinglydecreasing focal length.
 5. The storage medium of claim 1, wherein theposition of the highly detailed component of said successive images iscontrolled according to a control signal obtained by monitoring attitudeof a human eye.
 6. The storage medium of claim 1, having a head guidecontrol signal stored thereon in timed relation to said light controlsignal so as to be retrievable along with said light control signal withsuccessive images in said timed relation for inducing head attitudechanges in said viewer corresponding to head attitude changes of saidobserver.
 7. The storage medium of claim 6, wherein said head guidecontrol signal is produced by: (a) a head-mounted navigation system wornby a cameraman using said camera to produce said successive images, or(b) a mechanical head-coupled tracker worn by a cameraman using saidcamera to produce said successive images, or (c) a designer-selectedhead attitude signal source.
 8. A storage medium having a light controlsignal comprising successive images stored thereon, said signal storedon said storage medium comprising successive images having at least onefield of view provided from a camera that produces said signal havingsuccessive images of real objects therein, or provided from a computerfor creating said signal having successive computer generated imagesusing software, or provided from said computer on which post-productionwork is done on said successive images produced by said camera usingsoftware, said stored light control signal for later retrieval from saidstorage medium and presentation by a playback device of said successiveimages to a viewer, wherein selected successive images are presented asnonuniform images, each selected successive image comprising an area ina selected position within said at least one field of view that isemulative of the human fovea having nonuniform resolution, nonuniformdynamic range, a small colored area in an otherwise wide-field black andwhite image, nonuniform image concentration, nonuniform brightness, orother nonuniform informational content concentrated or highlightedtherein, in order to draw a gazing direction of the viewer thereto, andwherein said area of the field of view is changed in position within theat least one field of view between selected successive images such thatthe gazing direction of the viewer is drawn to move about within the atleast one field of view of the selected successive images in variousdirections so that the viewer experiences simulated active perceptspassively, through the eyes of the viewer, as if looking through theeyes of another person wherein said light control signal is produced bya computer workstation producing said successive images Of computergenerated images with software or by a camera producing said successiveimages of real objects and a workstation on which post-production workis done on said successive images of real objects from said camera inproducing said successive images.
 9. The storage medium of claim 8,having a head guide control signal stored thereon in timed relation tosaid light control signal so as to be retrievable along with saidsuccessive images in said timed relation for inducing head attitudechanges in said viewer corresponding to head attitude changes of saidother person.
 10. (canceled)
 11. The storage medium of claim 8, whereinthe successive images are for presentation to said eye of the viewer atvarious apparent distances for viewing by the eye of the viewer withcorresponding accommodation so as to focus on the successive images atthe various apparent distances.
 12. The storage medium of claim 8,wherein said successive images are of a scene and are for presentationat differing focal lengths for viewing with correspondingly differingaccommodation, wherein light is projected with details of said sceneover a widest field of view for a viewer accommodating at infinity andover correspondingly narrower fields of view for said vieweraccommodating at various distances closer than infinity, for providingsaid successive images formed for display with a largest sizecorresponding to said widest field of view and with decreasing sizescorresponding to increasingly narrower fields of view such thatparticular objects in said scene are viewable with increasinggranularity with decreasing focal length.
 13. A method comprising: (a)providing a light control signal comprising successive images having atleast one field of view, said light control signal produced by acomputer workstation producing said successive images of computergenerated images with software or a camera producing said successiveimages of real objects and a workstation on which post-production workis done on said successive images of real objects from said camera inproducing said successive images, and (b) storing said light controlsignal comprising said successive images having at least one field ofview on a storage medium, wherein said light control signal is for laterretrieval from said storage medium by a playback device for presentationby the playback device of said successive images having a field of viewto eyes of a viewer, wherein selected images of said successive imagescomprise successive nonuniform images, each nonuniform image having anarea in a selected position within said field of view with informationcontent concentrated or highlighted therein to simulate visualperceptions of a person actively viewing within said field of view, inorder to draw a gazing direction of the viewer thereto, and wherein saidarea of the field of view that has concentrated or highlightedinformational content is changed in position within the at least onefield of view between selected successive nonuniform images such that,upon said presentation by said playback device, the gazing direction ofthe viewer of said presentation is drawn to move about within the atleast one field of view of the selected successive nonuniform images invarious directions corresponding to said changes in position of theconcentrated or highlighted informational content so that the viewerperceives the successive nonuniform images as if looking through eyes ofsaid person.
 14. The method of claim 13, further comprising: (c)providing a head guide control signal indicative of head attitudechanges of said person in timed relation to said successive images ofsaid light control signal, and (d) storing said head guide controlsignal on said storage medium so as to be retrievable along with saidsuccessive images in said timed relation for inducing head attitudechanges in said viewer corresponding to said head attitude changes ofsaid person.
 15. The method of claim 14, wherein said head guide controlsignal is produced by (a) a head-mounted navigation system worn by acameraman using said camera to produce said successive images, or (b) amechanical head-coupled tracker worn by a cameraman using said camera toproduce said successive images, or (c) a designer-selected head attitudesignal source.
 16. The method of claim 13, wherein said successiveimages are of a scene and are for presentation at differing focallengths for viewing with correspondingly differing accommodation by saidviewer, wherein light is projected with details of said scene over awidest field of view for said viewer accommodating at infinity and overcorrespondingly narrower fields of view for said viewer accommodating atvarious distances closer than infinity, for providing said successiveimages formed for display with a largest size corresponding to saidwidest field of view and with decreasing sizes corresponding toincreasingly narrower fields of view such that particular objects insaid scene are viewable with correspondingly decreasing focal length.17. The method of claim 13, wherein the position of the area of thefield of view that has concentrated or highlighted informational contentof said successive images is controlled according to a control signalobtained by monitoring attitude of a human eye.
 18. The method of claim13, further comprising storing a head guide control signal on saidstorage medium in timed relation to said light control signal so as tobe retrievable along with said successive images in said timed relationfor inducing head attitude changes in said viewer corresponding to headattitude changes of said person.
 19. The method of claim 18, whereinsaid head guide control signal is produced by: (a) a head-mountednavigation system worn by a cameraman using said camera to produce saidsuccessive images, or (b) a mechanical head-coupled tracker worn by acameraman using said camera to produce said successive images, or (c) adesigner-selected head attitude signal source.
 20. The method of claim13, further comprising providing the light control signal for presentingthe successive images at various apparent distances for viewing by theeye of the viewer with corresponding accommodation so as to focus on thesuccessive images at the various apparent distances.